{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ac0b690450dbeacc",
   "metadata": {},
   "source": [
    "# DNN 网络识别手写图片\n",
    "本节仍属于 [PyTorch深度学习：深度神经网络（DNN）](https://www.bilibili.com/video/BV1Z8411R7PH) 的范畴，与之前的激活函数略有不同，故另起一页。\n",
    "\n",
    "此外，由于反复学习了一遍 MLP ，同时推荐学习唐一旦老师的 [徒手实现卷积神经网络--图像识别初体验以及模型过拟合](https://www.bilibili.com/video/BV1Gz421b7vb) ，\n",
    "此篇笔记由两位老师的讲解共同组成。\n",
    "\n",
    "唐一旦老师推荐一个模型的搭建过程可以是：初始化数据集、搭建模型网络、撰写评估函数、训练模型。之所以先撰写评估函数是为了检验之前撰写的代码准确性，与了解评估模型好坏的指标。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "initial_id",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-07-26T08:36:49.275306Z",
     "start_time": "2025-07-26T08:36:46.103772Z"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "from matplotlib_inline import backend_inline\n",
    "\n",
    "# 数据集相关\n",
    "from torchvision import datasets, transforms\n",
    "from torch.utils.data import Dataset, DataLoader"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e73b531883f962b8",
   "metadata": {},
   "source": [
    "接下来就要对图像进行张量化和归一化处理，具体的解释可以参考 [https://juejin.cn/post/7087397616731815966](https://juejin.cn/post/7087397616731815966)\n",
    "\n",
    "下面对两个过程进行简单的解释：\n",
    "\n",
    "## ToTensor 张量化\n",
    "\n",
    "这个函数会吧 PIL Image 对象转化为 Numpy 数组，具体来说他干了两个事情：\n",
    "\n",
    "- 将 (H x W x C) 格式的图片转化为 (C x H x W) ， HWC 格式的图片是以 高、宽、颜色 进行存储的，即读取的时候需要通过 image\\[h, w, c\\] 来获取图片像素值，而 CHW 格式存取时使用的是类似 image\\[c, h, w\\] 来获取图片像素值，前者是先定位高宽再取像素维度，后者是先取维度再取高宽。这样做的好处是，在读取的时候像素永远是同一个通道排在一起的，有助于神经网络理解同维度像素左右的联系。\n",
    "- 将像素值 \\[0, 255\\] 的区间映射为 \\[0.0, 1.0\\] 的区间，将大数据变小，对神经网络更加友好。\n",
    "\n",
    "## Normalize 归一化\n",
    "\n",
    "将 \\[\\~0.0, \\~1.0\\] 之间的数据变为 \\[\\~-1.0, \\~1.0\\] 之间，使其整个训练集的输入均值为 0 ，方差为 1，让神经网络中的 bias 可以在负数与正数内取值，接近原点，使梯度下降更高效，加快收敛速度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6207d2bf5115dae2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-07-26T08:36:49.441047Z",
     "start_time": "2025-07-26T08:36:49.279832Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA94AAADgCAYAAAD19b5rAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAIDVJREFUeJzt3XuYlmW1OOA1DAPDSQgEPAQoAiGKWxM8kIqmphaablHKDma2KxUzkjK9duluW2mKJzxu2yrs2lmhqD8rKxO9OiCIioUJEjgiaggiB5PzvL8/uqDYPM/A4LwMM3Pf18UfrOdb37tmmGe+b807PKuiKIoiAAAAgFK0auwCAAAAoDnTeAMAAECJNN4AAABQIo03AAAAlEjjDQAAACXSeAMAAECJNN4AAABQIo03AAAAlEjjDQAAACXSeJespqYmKioq4tprr22w53z88cejoqIiHn/88QZ7TthR7AnYnD0Bm7MnYHP2RPOg8U645557oqKiImbMmNHYpZTiiiuuiIqKii3+VFdXN3Zp7KSa+56IiHj11VfjzDPPjC5dusQuu+wSH/3oR2P+/PmNXRY7qZawJ/7Z8ccfHxUVFTF69OjGLoWdVHPfE3PmzIkxY8bEsGHDorq6OioqKqKmpqaxy2In1tz3RETEvffeG+9///ujuro6unfvHueee24sWbKkscvaabVu7AJoPLfddlt07Nhx098rKysbsRpoPG+//XYcc8wxsXz58rjsssuiqqoqrr/++hg+fHjMnDkzunXr1tglQqO5//77Y+rUqY1dBjSqqVOnxk033RSDBg2KfffdN2bOnNnYJUGjuu222+L888+PY489Nq677rpYuHBh3HjjjTFjxoyYNm2aG3oJGu8WbOTIkbHrrrs2dhnQ6G699daYO3duTJ8+PYYOHRoRESeddFLsv//+MW7cuPjOd77TyBVC41i9enVcfPHFcckll8Q3v/nNxi4HGs0pp5wSy5Yti06dOsW1116r8aZFW7t2bVx22WVx1FFHxa9//euoqKiIiIhhw4bFySefHHfeeWdceOGFjVzlzsevmm+ntWvXxje/+c04+OCDo3PnztGhQ4c48sgjY8qUKdmc66+/Pvr06RPt2rWL4cOHx6xZs7Z4zOzZs2PkyJHRtWvXqK6ujiFDhsRDDz201XreeeedmD17dr1+vaMoilixYkUURbHNOZDTlPfEpEmTYujQoZua7oiIgQMHxrHHHhs/+clPtpoPKU15T2z0ve99L2pra2Ps2LHbnAM5TXlPdO3aNTp16rTVx0F9NNU9MWvWrFi2bFmMGjVqU9MdETFixIjo2LFj3HvvvVu9Vkuk8d5OK1asiO9///tx9NFHx9VXXx1XXHFFLF68OE444YTkT0EnTpwYN910U1xwwQVx6aWXxqxZs+KDH/xgLFq0aNNjnn/++TjssMPihRdeiK9//esxbty46NChQ5x66qkxefLkOuuZPn167LvvvnHzzTdv88fQt2/f6Ny5c3Tq1Ck++clPblYL1FdT3RO1tbXxxz/+MYYMGbLF2iGHHBLz5s2LlStXbtsnAf5JU90TGy1YsCCuuuqquPrqq6Ndu3b1+tghpanvCWhoTXVPrFmzJiIi+drQrl27ePbZZ6O2tnYbPgMtTMEW7r777iIiiqeeeir7mPXr1xdr1qzZLPbWW28VPXv2LD772c9uir300ktFRBTt2rUrFi5cuCk+bdq0IiKKMWPGbIode+yxxeDBg4vVq1dvitXW1hbDhg0r+vfvvyk2ZcqUIiKKKVOmbBG7/PLLt/rx3XDDDcXo0aOLH/7wh8WkSZOKiy66qGjdunXRv3//Yvny5VvNp+Vpznti8eLFRUQU3/rWt7ZYu+WWW4qIKGbPnl3nc9DyNOc9sdHIkSOLYcOGbfp7RBQXXHDBNuXS8rSEPbHRNddcU0RE8dJLL9Urj5alOe+JxYsXFxUVFcW55567WXz27NlFRBQRUSxZsqTO52iJ3PHeTpWVldGmTZuI+Psds6VLl8b69etjyJAh8cwzz2zx+FNPPTX23HPPTX8/5JBD4tBDD42f//znERGxdOnSeOyxx+LMM8+MlStXxpIlS2LJkiXx5ptvxgknnBBz586NV199NVvP0UcfHUVRxBVXXLHV2i+66KIYP358nHXWWXH66afHDTfcEBMmTIi5c+fGrbfeWs/PBPxdU90Tq1atioiItm3bbrG28WCQjY+B+miqeyIiYsqUKXHffffFDTfcUL8PGurQlPcElKGp7oldd901zjzzzJgwYUKMGzcu5s+fH7/97W9j1KhRUVVVFRHeO6VovN+FCRMmxAEHHBDV1dXRrVu36N69e/zsZz+L5cuXb/HY/v37bxEbMGDAplEUf/nLX6IoivjGN74R3bt33+zP5ZdfHhERb7zxRmkfy1lnnRW77bZbPProo6Vdg+avKe6Jjb8mtfHXpv7Z6tWrN3sM1FdT3BPr16+PL33pS/GpT31qs3MPoCE0xT0BZWqqe+KOO+6ID3/4wzF27NjYZ5994qijjorBgwfHySefHBGx2eQk/s6p5tvpBz/4QXzmM5+JU089Nb761a9Gjx49orKyMr773e/GvHnz6v18G/8fxNixY+OEE05IPqZfv37vquat6dWrVyxdurTUa9B8NdU90bVr12jbtm28/vrrW6xtjO2xxx7v+jq0PE11T0ycODHmzJkTd9xxxxZzileuXBk1NTXRo0ePaN++/bu+Fi1LU90TUJamvCc6d+4cDz74YCxYsCBqamqiT58+0adPnxg2bFh07949unTp0iDXaU403ttp0qRJ0bdv37j//vs3O81v40+T/q+5c+duEXvxxRdjr732ioi/H3QWEVFVVRXHHXdcwxe8FUVRRE1NTRx00EE7/No0D011T7Rq1SoGDx4cM2bM2GJt2rRp0bdvXyfZsl2a6p5YsGBBrFu3Lj7wgQ9ssTZx4sSYOHFiTJ48OU499dTSaqB5aqp7AsrSHPZE7969o3fv3hERsWzZsnj66afj9NNP3yHXbmr8qvl2qqysjIjYbBTXtGnTYurUqcnHP/DAA5v9n4rp06fHtGnT4qSTToqIiB49esTRRx8dd9xxR/LO2+LFi+uspz4jMVLPddttt8XixYvjxBNP3Go+pDTlPTFy5Mh46qmnNmu+58yZE4899licccYZW82HlKa6Jz72sY/F5MmTt/gTEfHhD384Jk+eHIceemidzwEpTXVPQFma25649NJLY/369TFmzJjtym/u3PGuw1133RWPPPLIFvGLLrooRowYEffff3+cdtpp8ZGPfCReeumluP3222PQoEHx9ttvb5HTr1+/OOKII+K8886LNWvWxA033BDdunWLr33ta5sec8stt8QRRxwRgwcPjn/7t3+Lvn37xqJFi2Lq1KmxcOHCeO6557K1Tp8+PY455pi4/PLLt3ogQp8+fWLUqFExePDgqK6ujt/97ndx7733xoEHHhhf+MIXtv0TRIvTXPfE+eefH3feeWd85CMfibFjx0ZVVVVcd9110bNnz7j44ou3/RNEi9Mc98TAgQNj4MCBybW9997bnW7q1Bz3RETE8uXLY/z48RER8fvf/z4iIm6++ebo0qVLdOnSJUaPHr0tnx5aoOa6J6666qqYNWtWHHroodG6det44IEH4le/+lVceeWVzgfJ2fEHqe/8Nh7/n/vzyiuvFLW1tcV3vvOdok+fPkXbtm2Lgw46qHj44YeLs88+u+jTp8+m59p4/P8111xTjBs3rujVq1fRtm3b4sgjjyyee+65La49b9684tOf/nSx2267FVVVVcWee+5ZjBgxopg0adKmx7zbkRif+9znikGDBhWdOnUqqqqqin79+hWXXHJJsWLFinfzaaMZa+57oiiK4pVXXilGjhxZ7LLLLkXHjh2LESNGFHPnzt3eTxnNXEvYE/9XGCdGHZr7nthYU+rPP9cOGzX3PfHwww8XhxxySNGpU6eiffv2xWGHHVb85Cc/eTefsmavoij+6XcbAAAAgAbl/3gDAABAiTTeAAAAUCKNNwAAAJRI4w0AAAAl0ngDAABAiTTeAAAAUCKNNwAAAJSo9bY+8PhWZ5RZBzSKX9f+dLtz7QmaI3sCtrS9+8KeoDnyOgFb2pZ94Y43AAAAlEjjDQAAACXSeAMAAECJNN4AAABQIo03AAAAlEjjDQAAACXSeAMAAECJNN4AAABQIo03AAAAlEjjDQAAACXSeAMAAECJNN4AAABQIo03AAAAlEjjDQAAACXSeAMAAECJNN4AAABQIo03AAAAlEjjDQAAACXSeAMAAECJNN4AAABQIo03AAAAlEjjDQAAACXSeAMAAECJNN4AAABQIo03AAAAlKh1YxcAsL3Wf/Dg7Nrr569Jxp87fEI251+mnp2M73FLm2xO5ZRnsmsAABDhjjcAAACUSuMNAAAAJdJ4AwAAQIk03gAAAFAijTcAAACUyKnmJatonf4UV3bftUGvM2fsXsn4hva12Zw++7yRjLc/vyKb89fr0qc7PzPkx9mcJRv+lowf+tOLszn9vvJkdo2Wp3b4Qcn4TXfdnM3pV5Xee/kdEfHs4Xcn43OGbMjmfHWvw+p4RmiZ/jby0GT86u/dls35zzM/nYwXM2Y1SE3QEOZdc3gy/sJZ+dejqorKZPyo8z+fzWn3wPT6FQbs9NzxBgAAgBJpvAEAAKBEGm8AAAAokcYbAAAASqTxBgAAgBJpvAEAAKBELXacWOW+/ZPxom1VNue14V2S8VWHpcdlRUR07Zxe++2/5Mdv7Si/eKdTMn71zSdmc6YN/t9k/KV1q7I5Vy06Phnf47dFHdXR0qz70JDs2tdu/Z9kfEBVerxdRERtZnDY/HXrsjnLa9sm4welwxERseakocl4uyl/yte2enX+CdmqVR89JL/WLT22p+tdU8sqh4Q3hqR/rv+fNSfv4Eqg/v46Zlh27fFR30vG1xX516Msb4OgRXHHGwAAAEqk8QYAAIASabwBAACgRBpvAAAAKJHGGwAAAErUrE8133D0+7Nr191zSzJe1ynJTdG6YkN27ZvjP5OMt/5b/pjNw386Ohnv9Or6bE7bJekTz9vPmJbNoWmr3GWX7NrfjhqYjI+5Pn1ifkTEMe3ezqzU/2eH97yVP632N7cenoz//oqbsjm//v7tyfigH6T3SkRE30ucsP1uvHZU/t+9/T7L0gt3lVNLi9YqfYJ8RETRO/19/9ges7M5v6nI703Ykd7ulZ6KERHRtVXzep9I07D2hPzkl5c/kf56Pe/9T2RzvvyeF+tdw+DvX5hda/96undYNmxNNqfPD9Ov5W1+OaN+hTUh7ngDAABAiTTeAAAAUCKNNwAAAJRI4w0AAAAl0ngDAABAiTTeAAAAUKJmPU6s7ZzXsmtPr+6VjA+oWlRWOdvs4tcPS8bnv71rNueefSYl48tr86PBet70h/oVtp3yFdBcLZy4Z3btqaHpUX47yrd6PJVde6RjepzROTUfyuZM2OvRZHyXQW/WrzC22X+M+Gl27eoX8v9WNKzKffpk12YPT89vO3D6J7M5ezz1p3ddE9TH22ccmozfd9qNdWRVJKO3L0uPyoyIePTM9CioDi8/n83JDzSjOVv8xfRY0/Ffy793GtI2PTq4VR33V8+uOS67dlDnBcn4c5+ra1+k1VXDsK4fT8a7/rLel2ky3PEGAACAEmm8AQAAoEQabwAAACiRxhsAAABKpPEGAACAEjXrU83Xv/7X7Nr4q89Ixr994t+yOZV/7JiMP3f++PoVFhFXLjkgu/aX49on4xuWvZ7NOevw85Pxmi/la9g7nssvwjZY/8GDk/EfHXhzNqdVtKn3dc55+dhkfMaj+2Zz/nRuuoYpq6qzOT1mrErG//JW/rTaqu9MScZbpQ++pQFUVaxv7BKIiNbff6feOavm7VJCJZC3esQh2bXLv5s+fX9AVf2/gU+488Ts2m5/3jFTZNi5VFSl3++sPu5fsjn3XXpNMr5H67bZnHNfPj4Zf/na92VzOvxsZnZtSvveyfgTkwdkc+7r/1B2LWfFzG7JeNd6P1PT4Y43AAAAlEjjDQAAACXSeAMAAECJNN4AAABQIo03AAAAlEjjDQAAACVq1uPE6tL17qnJePf/lz7aPiJiw5tLk/H99v9sNuf5o9KjKh76r+HZnB7L6j92omJqejTY3ukPE7ZZ7fCDsms33ZUe2dWvKv+tpTZqk/FTZp+WzakcmR7z1+UjRTZn0P+MTsYH3PJKNqfVK88m4+/5bTYl1n17QzJ+3wHpvR8R8dlj0nP+Kqc8k79QC1R7xIHJ+JHVv9uxhZC0V4c3653T69H0foGyvP7J1dm1Y9rl1iqzOWfXHJeM73ajkWFs7vXRQ5Lx6WNvrCMrPTbsjL+cnM1Yf/q6ZLz9kmnZnPy7p4jXPp8eFTutf111p/3inU7ZtX53pN+PNeeBoe54AwAAQIk03gAAAFAijTcAAACUSOMNAAAAJdJ4AwAAQIla7KnmORuW1P+U1nUr2tQ7Z79P/Dm7tvi2zGmatU6DpTwVB++XjC/5yqpszoCq9Nf+02vy13ns7UHJ+Jv39srmdHsrfTx/5x88mc3pnInvqNMye1amTyaNiHjzy+8k4z2mlFVN0/TyiHbJeI/K9ju4kpat9V69k/GRXR+q93O1e+mt7JpXON6N1u/dMxl//si7sznrivRX3QvpQ6IjImLBdQOS8Q6RP0Ga5mvu+EOza3P+dXwynp7t8nf7/vqLyfjAsTXZnO3pXeryxfMebLDnuvLbZ2fX3vNKyxu95I43AAAAlEjjDQAAACXSeAMAAECJNN4AAABQIo03AAAAlEjjDQAAACUyTqwB7HvJi9m1cwYfm4zf3ec32ZzhZ1yQjHf6cX50EmyLVu3zY5jWf29FMv7kwPuzOS+tX5uMf+Wyi7M57/ntgmS8R4c3sjnNbczQIbu/nIzX7Ngydnqt+62sd87q2V0avpAW7pUbOiTjH2ibH4rz3yvem15Ylv4+A9uicr/3ZdeG/O+sBrvOqPu/lF3b5z7vxVqieeMOS8bn/Ost2ZzltauT8TNmn5XNed+F6Z5iw8r6vx626pD+3h0R8ebIA7JrH+14Tfr5Ij3iMyJi4E/TvUu/e1reyLC6uOMNAAAAJdJ4AwAAQIk03gAAAFAijTcAAACUSOMNAAAAJXKqeQPYsGx5du3N8/ZNxhc8tCqb8/UrJybjl555WjaneLZzMt7r23WcJlgU+TWapVXD98uu/XLgrfV+vs9dNCYZ7/RA/tTX9fW+Cmy7HjPyJ223FJW7dsuuLTp9QDLe9cyF2ZwnBvx3ZqU6m3PbLacm4z0W/SGbA1vz8in5r+1J3Z7NrFRmc86ad3IyPuCqedmc5jZlg3+o7NkjuzbhtPR7pNrIv+bkTi9vc3x6ssnfn6/+Wh04KBnf/64XsjlX9rypjmdsm4x+YObHshnvuyJ9Lftlc+54AwAAQIk03gAAAFAijTcAAACUSOMNAAAAJdJ4AwAAQIk03gAAAFAi48RKVvtc+nj9j/3HV7M5P7z82mR85mHpMWMREXFYOrxfh9HZlP53vp6Mr59fk78OTdoB/zkzu9Yq83O4c14+NpvT7oHp77akJq+qIj2qZl0d0/oqK4zyK8uqrumv4w4NfJ3aIw9KxovKimzOK8elR7Ss3WNdNqdVm/Qwll8dOT6bU5UvIf66IV3DN+bnx1UurU0PuGnfKj8opue0lcm4r3y2xdJzDk/GJ3/xmjqyqpLRL74yPJux7uz0ftiweEEd16G5qqhOfz1ERAxpW//BWO2+1CZ9nT69sjlzv/jeZPxDxz2TzRnT47+S8d6t22Vz6hpbtiEzbrjix7vmc5bNreMZ2cgdbwAAACiRxhsAAABKpPEGAACAEmm8AQAAoEQabwAAACiRU80bSde7pmbXRs+5IBnf5aqF2Zwf9f1lMv78p2/O5gzs9blk/H3/kf95zIa587Nr7DyWfSp9Iuy/90yfmB8RURvp0zef/tWgbE7v+EP9CmuG1hXpk05r6zgz9JEX0p/T/pE/tbQlWrM6fUpxbR1nY9992fXJ+EOjD2yIkja5pNv3k/FWkT9SfFWxNhl/bUP+tNybFx+djB/36JezOV2eTe/liIjdf7UoGa94Of/6sviF9Mm4PSvzp7EXT/0puwYREZX7vS+79ocrc+9dqut9nakL98qu9aqZVe/no/kqVq/Jrk1bk349OrRt/vvgg4/em4zX9f5gezy6Kn3a+Nw6xqsc0+7t7NqMtenXkC4T870L28YdbwAAACiRxhsAAABKpPEGAACAEmm8AQAAoEQabwAAACiRxhsAAABKZJzYTqji9zOT8XdG9sjmDB11YTI+7ZIbszmzj0mPw/nEXh/K5iw/IrvETmR9evpPdG6VHzM0dXXbZLzvxNfy16lXVTu/Vu3bJ+Ozr92/jqynk9FPzD8pmzHwopeS8fxQqZap3yefTcb3++7obE6voa+WVc5mprwxIBlf/Iv3ZnO6PZ8eO9PmkafquFI6Z0DMqCMnL/c19uolw7I5Q9umR8jc+/ae21UDRES8eFn6+21Efkzj9uh9VX4tP2yJlmjDojeya5eflx7Be+3tt2ZzDsi85frBil7ZnCufOCUZH3DP6mxO60XLk/EeP1qazTmm12PZtbOnpD/W7X3d4R/c8QYAAIASabwBAACgRBpvAAAAKJHGGwAAAEqk8QYAAIASOdW8CanrtMWeN6XXVn8tf+50+4r0cYt37vVwNmfEaV9OP9fkadkcmoY3N3RMxtfPr9mxhZQsd3J5RMScqwYn47M/enM25xfvdE7GX7ulXzan01tPZtfYur0vTZ+yvTPYPRY0dgnbpf1Ri+ud8+9TTs+uDYjp76YcmpHa4Qcl41cOeaBBr3P8rI8l4x1nzGrQ69Aytfll+kTvy/Y+pEGvsz3fO1d+NF3Dz3o/mM1ZV+TvvbaryU/A4d1xxxsAAABKpPEGAACAEmm8AQAAoEQabwAAACiRxhsAAABKpPEGAACAEhknthOqPeLAZHzeGdXZnP0PrEnGcyPD6jJ+aXr0R0RE+wfT4xRo+sb+/oxkfEA8vYMraRi5ETZvfGVVNueFIemxYcf+aVQ2p8OJ85PxTmFkGM1bnweLxi6BJuDb9/xXMr5/Vf2/fsa+flR2rfPH30rGN9T7KtC0rG+Xvo+6rsh/9ddGbXZt73vSYzHzA4rZVu54AwAAQIk03gAAAFAijTcAAACUSOMNAAAAJdJ4AwAAQImcal6yiiH7J+Mvfil/2vidH5iQjB9VvbZBatpoTbEuGX9y6d75pNrXG7QGSlKRDreq42dtNx7xo2T8lhjQEBWV4uVvHZ5du+/T1yXjA6rye+/9089Oxvc47c/1KwyAiIg4qE39T1zOmXr3+7NrPd76Q72fD5qDTvdmpqiM27F1sHXueAMAAECJNN4AAABQIo03AAAAlEjjDQAAACXSeAMAAECJNN4AAABQIuPE6qH13n2S8Xnn7JHNuWLUvcn46R2XNEhNW3PZoiHZtSduPCwZf8+EqWWVw45SpMO1UZtNGd7uzWT8y/ccnM3Z5+7081X9dWU2Z9Hw7sl411ELszkX9v5NMn5S+6ezOQ/9rWcy/uk/nZjN2fWODtk1aKkqK9I/o39rQFU2Z7dflFUNO6NXJqVHp0ZEVFXMbLDr7P54/r1T/YeTQfOw8mPp9/MR+fdINA53vAEAAKBEGm8AAAAokcYbAAAASqTxBgAAgBJpvAEAAKBELfZU89Z79U7Glx+8ezZn1LceSca/2OX+Bqlpay5+PXdqYcTUW9Onl3e9Z3o25z21Ti/nH6or0t8OXjj+9mzO746sTsbnrtktm3NO55p61VWXi147Mrv2yB8OTMb7X/Rkg10fWoINRWYagh/dtzi1ww9Kxm848AfZnHVF+rzx5bWrszlDf/HlZHzgy3/OFwct1PK+vhk3Ff6lAAAAoEQabwAAACiRxhsAAABKpPEGAACAEmm8AQAAoEQabwAAAChRsxgn1nr39OiipXd1yOact/cTyfjHOy1qkJq2ZvSrR2TXnrntwGR810mzsjldVxoNxj/0fPyNZPySLxyezbl6t/p/DR1VvTYZP6K6pt7P9eya/M8BP/7E55PxAec8nc3pH8aGQZneGfpOY5fADra6a5tk/Ijqv9WRVZmM/vKd9FjXiIgBn38qGc8MtoMWbc8n0t+Lq0an915ExLqirGqoizveAAAAUCKNNwAAAJRI4w0AAAAl0ngDAABAiTTeAAAAUKKd7lTztScMScfHLM3mXNbv58n4h9rVdcpmw1m0YVV27aiHLk7GB/777GxO12Xp06Wd5sm22vDivGR87hl7ZXMGXXhhMv7nM8c3REmbDPz5+cn4+27Nn5A84Nn86eVAuSor/IweYGdV8fuZyfg9K3pkcz7e6dXs2jv77Z6Mt3llYb3qYkteTQEAAKBEGm8AAAAokcYbAAAASqTxBgAAgBJpvAEAAKBEGm8AAAAo0U43Tqzm1PTPAl4c/NMGvc4ty/ZJxm984kPZnIoNFcn4wCtfyub0XzQtGd9QR21QlvXza7Jr/cak104ZM7RBaxgQTyXjRYNeBaiPNY92z65tONAwS/5ul5l/TcYvXPjBbM7tvZ4oqxygDtffMTK79vGxN2bXdv/GX5LxN5cdkL/Yk3/c5rpaMne8AQAAoEQabwAAACiRxhsAAABKpPEGAACAEmm8AQAAoEQVRVFs02HCx7c6o+xaYIf7de32n5ZvT9Ac2ROwpe3dF/YEzZHXiaahctdu2bU29+UHW/2438PJ+PDnPp7N6XrW4mR8w7Ll2ZzmZlv2hTveAAAAUCKNNwAAAJRI4w0AAAAl0ngDAABAiTTeAAAAUCKNNwAAAJQof5Y8AAAATc6GJW9m19aenh81tu+4LyTjLxx3RzbnlIHnphee/GM2pyVyxxsAAABKpPEGAACAEmm8AQAAoEQabwAAACiRxhsAAABK5FRzAACAFqKuE8/7n51eOyWG1vGMTi/fFu54AwAAQIk03gAAAFAijTcAAACUSOMNAAAAJdJ4AwAAQIk03gAAAFCiiqIoisYuAgAAAJord7wBAACgRBpvAAAAKJHGGwAAAEqk8QYAAIASabwBAACgRBpvAAAAKJHGGwAAAEqk8QYAAIASabwBAACgRP8fTXXPDSLQw5wAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 1000x400 with 5 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 转化为张量与归一化处理\n",
    "# transform = transforms.Compose(  # 将多个图像变换操作组合在一起\n",
    "#     [\n",
    "#         transforms.ToTensor(),\n",
    "#         transforms.Normalize((0.1307,), (0.3081,))\n",
    "#     ]\n",
    "# )\n",
    "\n",
    "# 下载训练集\n",
    "train_Data_raw = datasets.MNIST(\n",
    "    root='./datasets/mnist/',\n",
    "    train=True,\n",
    "    download=True,\n",
    "    transform=None\n",
    ")\n",
    "test_Data_raw = datasets.MNIST(\n",
    "    root='./datasets/mnist/',\n",
    "    train=False,\n",
    "    download=True,\n",
    "    transform=None\n",
    ")\n",
    "\n",
    "# 展示 5 张照片\n",
    "plt.figure(figsize=(10, 4))\n",
    "for _ in range(5):\n",
    "    img, label = train_Data_raw[_]\n",
    "    plt.subplot(1, 5, _ + 1)\n",
    "    plt.imshow(img)  # cmap=plt.cm.binary 色板颜色设置为黑白\n",
    "    plt.title(f\"Label: {label}\")\n",
    "    plt.axis('off')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()  # 展示彩色是因为 plt 是用于数学绘图的，这样是把所有颜色值一样的像素给到不同的高度的颜色了（参考等高图）\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ca8ddc0eff89b28c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-07-26T08:36:54.950355Z",
     "start_time": "2025-07-26T08:36:49.557788Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0118, 0.0706, 0.0706, 0.0706,\n",
      "          0.4941, 0.5333, 0.6863, 0.1020, 0.6510, 1.0000, 0.9686, 0.4980,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.1176, 0.1412, 0.3686, 0.6039, 0.6667, 0.9922, 0.9922, 0.9922,\n",
      "          0.9922, 0.9922, 0.8824, 0.6745, 0.9922, 0.9490, 0.7647, 0.2510,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.1922,\n",
      "          0.9333, 0.9922, 0.9922, 0.9922, 0.9922, 0.9922, 0.9922, 0.9922,\n",
      "          0.9922, 0.9843, 0.3647, 0.3216, 0.3216, 0.2196, 0.1529, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0706,\n",
      "          0.8588, 0.9922, 0.9922, 0.9922, 0.9922, 0.9922, 0.7765, 0.7137,\n",
      "          0.9686, 0.9451, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.3137, 0.6118, 0.4196, 0.9922, 0.9922, 0.8039, 0.0431, 0.0000,\n",
      "          0.1686, 0.6039, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0549, 0.0039, 0.6039, 0.9922, 0.3529, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.5451, 0.9922, 0.7451, 0.0078, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0431, 0.7451, 0.9922, 0.2745, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.1373, 0.9451, 0.8824, 0.6275,\n",
      "          0.4235, 0.0039, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.3176, 0.9412, 0.9922,\n",
      "          0.9922, 0.4667, 0.0980, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.1765, 0.7294,\n",
      "          0.9922, 0.9922, 0.5882, 0.1059, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0627,\n",
      "          0.3647, 0.9882, 0.9922, 0.7333, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.9765, 0.9922, 0.9765, 0.2510, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.1804, 0.5098,\n",
      "          0.7176, 0.9922, 0.9922, 0.8118, 0.0078, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.1529, 0.5804, 0.8980, 0.9922,\n",
      "          0.9922, 0.9922, 0.9804, 0.7137, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0941, 0.4471, 0.8667, 0.9922, 0.9922, 0.9922,\n",
      "          0.9922, 0.7882, 0.3059, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0902, 0.2588, 0.8353, 0.9922, 0.9922, 0.9922, 0.9922, 0.7765,\n",
      "          0.3176, 0.0078, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0706, 0.6706,\n",
      "          0.8588, 0.9922, 0.9922, 0.9922, 0.9922, 0.7647, 0.3137, 0.0353,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.2157, 0.6745, 0.8863, 0.9922,\n",
      "          0.9922, 0.9922, 0.9922, 0.9569, 0.5216, 0.0431, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.5333, 0.9922, 0.9922, 0.9922,\n",
      "          0.8314, 0.5294, 0.5176, 0.0627, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000]]])\n",
      "tensor([[[-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.3860, -0.1951,\n",
      "          -0.1951, -0.1951,  1.1795,  1.3068,  1.8032, -0.0933,  1.6887,\n",
      "           2.8215,  2.7197,  1.1923, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.0424,  0.0340,  0.7722,  1.5359,  1.7396,  2.7960,\n",
      "           2.7960,  2.7960,  2.7960,  2.7960,  2.4396,  1.7650,  2.7960,\n",
      "           2.6560,  2.0578,  0.3904, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "           0.1995,  2.6051,  2.7960,  2.7960,  2.7960,  2.7960,  2.7960,\n",
      "           2.7960,  2.7960,  2.7960,  2.7706,  0.7595,  0.6195,  0.6195,\n",
      "           0.2886,  0.0722, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.1951,  2.3633,  2.7960,  2.7960,  2.7960,  2.7960,  2.7960,\n",
      "           2.0960,  1.8923,  2.7197,  2.6433, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242,  0.5940,  1.5614,  0.9377,  2.7960,  2.7960,  2.1851,\n",
      "          -0.2842, -0.4242,  0.1231,  1.5359, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.2460, -0.4115,  1.5359,  2.7960,  0.7213,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242,  1.3450,  2.7960,  1.9942,\n",
      "          -0.3988, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.2842,  1.9942,  2.7960,\n",
      "           0.4668, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,  0.0213,  2.6433,\n",
      "           2.4396,  1.6123,  0.9504, -0.4115, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,  0.6068,\n",
      "           2.6306,  2.7960,  2.7960,  1.0904, -0.1060, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "           0.1486,  1.9432,  2.7960,  2.7960,  1.4850, -0.0806, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.2206,  0.7595,  2.7833,  2.7960,  1.9560, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242,  2.7451,  2.7960,  2.7451,  0.3904,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "           0.1613,  1.2305,  1.9051,  2.7960,  2.7960,  2.2105, -0.3988,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,  0.0722,  1.4596,\n",
      "           2.4906,  2.7960,  2.7960,  2.7960,  2.7578,  1.8923, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.1187,  1.0268,  2.3887,  2.7960,\n",
      "           2.7960,  2.7960,  2.7960,  2.1342,  0.5686, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.1315,  0.4159,  2.2869,  2.7960,  2.7960,  2.7960,\n",
      "           2.7960,  2.0960,  0.6068, -0.3988, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.1951,\n",
      "           1.7523,  2.3633,  2.7960,  2.7960,  2.7960,  2.7960,  2.0578,\n",
      "           0.5940, -0.3097, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242,  0.2758,  1.7650,  2.4524,\n",
      "           2.7960,  2.7960,  2.7960,  2.7960,  2.6815,  1.2686, -0.2842,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242,  1.3068,  2.7960,  2.7960,\n",
      "           2.7960,  2.2742,  1.2941,  1.2559, -0.2206, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242]]])\n",
      "tensor([[[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.3294, 0.7255,\n",
      "          0.6235, 0.5922, 0.2353, 0.1412, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.8706, 0.9961,\n",
      "          0.9961, 0.9961, 0.9961, 0.9451, 0.7765, 0.7765, 0.7765, 0.7765,\n",
      "          0.7765, 0.7765, 0.7765, 0.7765, 0.6667, 0.2039, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.2627, 0.4471,\n",
      "          0.2824, 0.4471, 0.6392, 0.8902, 0.9961, 0.8824, 0.9961, 0.9961,\n",
      "          0.9961, 0.9804, 0.8980, 0.9961, 0.9961, 0.5490, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0667, 0.2588, 0.0549, 0.2627, 0.2627,\n",
      "          0.2627, 0.2314, 0.0824, 0.9255, 0.9961, 0.4157, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.3255, 0.9922, 0.8196, 0.0706, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0863, 0.9137, 1.0000, 0.3255, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.5059, 0.9961, 0.9333, 0.1725, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.2314, 0.9765, 0.9961, 0.2431, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.5216, 0.9961, 0.7333, 0.0196, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0353,\n",
      "          0.8039, 0.9725, 0.2275, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.4941,\n",
      "          0.9961, 0.7137, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.2941, 0.9843,\n",
      "          0.9412, 0.2235, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0745, 0.8667, 0.9961,\n",
      "          0.6510, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0118, 0.7961, 0.9961, 0.8588,\n",
      "          0.1373, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.1490, 0.9961, 0.9961, 0.3020,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.1216, 0.8784, 0.9961, 0.4510, 0.0039,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.5216, 0.9961, 0.9961, 0.2039, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.2392, 0.9490, 0.9961, 0.9961, 0.2039, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.4745, 0.9961, 0.9961, 0.8588, 0.1569, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.4745, 0.9961, 0.8118, 0.0706, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000],\n",
      "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
      "          0.0000, 0.0000, 0.0000, 0.0000]]])\n",
      "tensor([[[-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,  0.6450,\n",
      "           1.9305,  1.5996,  1.4978,  0.3395,  0.0340, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,  2.4015,\n",
      "           2.8088,  2.8088,  2.8088,  2.8088,  2.6433,  2.0960,  2.0960,\n",
      "           2.0960,  2.0960,  2.0960,  2.0960,  2.0960,  2.0960,  1.7396,\n",
      "           0.2377, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,  0.4286,\n",
      "           1.0268,  0.4922,  1.0268,  1.6505,  2.4651,  2.8088,  2.4396,\n",
      "           2.8088,  2.8088,  2.8088,  2.7578,  2.4906,  2.8088,  2.8088,\n",
      "           1.3577, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.2078,  0.4159, -0.2460,\n",
      "           0.4286,  0.4286,  0.4286,  0.3268, -0.1569,  2.5797,  2.8088,\n",
      "           0.9250, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242,  0.6322,  2.7960,  2.2360,\n",
      "          -0.1951, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.1442,  2.5415,  2.8215,  0.6322,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242,  1.2177,  2.8088,  2.6051,  0.1358,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242,  0.3268,  2.7451,  2.8088,  0.3649, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242,  1.2686,  2.8088,  1.9560, -0.3606, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.3097,  2.1851,  2.7324,  0.3140, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242,  1.1795,  2.8088,  1.8923, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "           0.5304,  2.7706,  2.6306,  0.3013, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.1824,\n",
      "           2.3887,  2.8088,  1.6887, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.3860,  2.1596,\n",
      "           2.8088,  2.3633,  0.0213, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,  0.0595,  2.8088,\n",
      "           2.8088,  0.5559, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.0296,  2.4269,  2.8088,\n",
      "           1.0395, -0.4115, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242,  1.2686,  2.8088,  2.8088,\n",
      "           0.2377, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242,  0.3522,  2.6560,  2.8088,  2.8088,\n",
      "           0.2377, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242,  1.1159,  2.8088,  2.8088,  2.3633,\n",
      "           0.0849, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242,  1.1159,  2.8088,  2.2105, -0.1951,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
      "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
      "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242]]])\n"
     ]
    }
   ],
   "source": [
    "# 手动进行张量化和归一化\n",
    "class DatasetMNIST(Dataset):\n",
    "    def __init__(self, raw):\n",
    "        super().__init__()\n",
    "        self.raw = raw\n",
    "        self.imgs = []\n",
    "        self.labels = []\n",
    "\n",
    "        self.ToTensor = transforms.ToTensor()\n",
    "        self.Normalize = transforms.Normalize((0.1307,), (0.3081,))\n",
    "\n",
    "        # 其实可以一把写完，但是这里为了展示中间过程所以拆开\n",
    "        for img, label in self.raw:\n",
    "            self.imgs.append(self.ToTensor(img))\n",
    "            self.labels.append(label)\n",
    "\n",
    "        print(self.imgs[0])\n",
    "\n",
    "        for i in range(len(self.imgs)):\n",
    "            self.imgs[i] = self.Normalize(self.imgs[i])\n",
    "\n",
    "        print(self.imgs[0])\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        return self.imgs[index], self.labels[index]\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.imgs)\n",
    "\n",
    "\n",
    "train_Data = DatasetMNIST(train_Data_raw)\n",
    "test_Data = DatasetMNIST(test_Data_raw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d1fc7f7600426bac",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-07-26T08:36:54.969256Z",
     "start_time": "2025-07-26T08:36:54.965051Z"
    }
   },
   "outputs": [],
   "source": [
    "# 构建神经网络\n",
    "class net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(net, self).__init__()\n",
    "        self.net = nn.Sequential(\n",
    "            nn.Flatten(),  # 把输入摊平\n",
    "            nn.Linear(28 * 28, 512), nn.ReLU(),\n",
    "            nn.Linear(512, 256), nn.ReLU(),\n",
    "            nn.Linear(256, 128), nn.ReLU(),\n",
    "            nn.Linear(128, 64), nn.ReLU(),\n",
    "            nn.Linear(64, 10)\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.net(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f0c08a3a437de955",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-07-26T08:36:54.986215Z",
     "start_time": "2025-07-26T08:36:54.982804Z"
    }
   },
   "outputs": [],
   "source": [
    "# 批处理数据集\n",
    "train_loader = DataLoader(train_Data, batch_size=64, shuffle=True)\n",
    "test_loader = DataLoader(test_Data, batch_size=64, shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c1ae693d852b1c04",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-07-26T08:36:55.152702Z",
     "start_time": "2025-07-26T08:36:54.993676Z"
    }
   },
   "outputs": [],
   "source": [
    "# 初始化训练参数\n",
    "net = net().to('cuda')\n",
    "\n",
    "epoch = 10\n",
    "learning_rate = 0.001\n",
    "\n",
    "loss_fn = nn.CrossEntropyLoss()  # 损失器\n",
    "optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)  # 优化器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e5f57393e1ef6ad0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-07-26T08:37:03.902445Z",
     "start_time": "2025-07-26T08:36:55.159237Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAGwCAYAAABB4NqyAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAO5tJREFUeJzt3Xl8VPW9//H3zCSZ7DvZSCABZJUAAYws1rZGwWqtt7aC1YJQ9Xet2mpaFWwFW6xBRS8PLwitV1zaa8WlolctV80VFAyiBATZQSSBkIQEMpOFbDPz+yNkICVAyHZmeT0fj3kkOTlz8pnOxXnf7/mczzG5XC6XAAAA/IjZ6AIAAAB6GwEIAAD4HQIQAADwOwQgAADgdwhAAADA7xCAAACA3yEAAQAAvxNgdAGeyOl0qqSkRBERETKZTEaXAwAAOsDlcqm6ulopKSkym8+9xkMAakdJSYnS0tKMLgMAAHRCcXGxUlNTz7kPAagdERERklr+B4yMjDS4GgAA0BF2u11paWnuz/FzIQC1o/W0V2RkJAEIAAAv05H2FZqgAQCA3yEAAQAAv0MAAgAAfocABAAA/A4BCAAA+B0CEAAA8DsEIAAA4HcIQAAAwO8QgAAAgN8hAAEAAL9DAAIAAH6HAAQAAPwOAagXuVwuFVXW6XDVCaNLAQDAr3lEAFq6dKnS09MVHBys7Oxsbdy48az7Pvfcc7rssssUExOjmJgY5eTknLH/rbfeKpPJ1OYxderUnn4Z5/Wn93bqO09+rBfXHzC6FAAA/JrhAWjlypXKzc3V/PnzVVhYqFGjRmnKlCkqLy9vd/81a9bopptu0scff6yCggKlpaXpqquu0uHDh9vsN3XqVB05csT9+Pvf/94bL+echiZHSpIKi6qMLQQAAD9ncrlcLiMLyM7O1vjx47VkyRJJktPpVFpamu655x7NmTPnvM93OByKiYnRkiVLNGPGDEktK0BVVVVatWpVh2poaGhQQ0OD+2e73a60tDTZbDZFRkZe+Is6i2+O1uj7T61VUIBZXz8yRUEBhudPAAB8ht1uV1RUVIc+vw39BG5sbNSmTZuUk5Pj3mY2m5WTk6OCgoIOHaOurk5NTU2KjY1ts33NmjVKSEjQkCFDdOedd6qysvKsx8jLy1NUVJT7kZaW1rkXdB4Z8WGKCQ1UY7NT20tsPfI3AADA+RkagCoqKuRwOJSYmNhme2JiokpLSzt0jAcffFApKSltQtTUqVP18ssvKz8/X48//rjWrl2rq6++Wg6Ho91jzJ07Vzabzf0oLi7u/Is6B5PJpDH9YiRxGgwAACMFGF1AVyxcuFCvvvqq1qxZo+DgYPf26dOnu78fOXKkMjMzNXDgQK1Zs0ZXXHHFGcexWq2yWq29UnNWv2j9365yFRYd1y+U0St/EwAAtGXoClB8fLwsFovKysrabC8rK1NSUtI5n7to0SItXLhQH3zwgTIzM8+574ABAxQfH699+/Z1ueauyjq5ArT54HGDKwEAwH8ZGoCCgoI0duxY5efnu7c5nU7l5+drwoQJZ33eE088oQULFmj16tUaN27cef/OoUOHVFlZqeTk5G6puytGpUXLbJJKbPUqtdUbXQ4AAH7J8MuQcnNz9dxzz+mll17Szp07deedd6q2tlazZs2SJM2YMUNz58517//444/r4Ycf1ooVK5Senq7S0lKVlpaqpqZGklRTU6P7779fGzZs0Lfffqv8/Hz96Ec/0qBBgzRlyhRDXuPpwqwBGpLUejk8q0AAABjB8AA0bdo0LVq0SPPmzdPo0aO1ZcsWrV692t0YXVRUpCNHjrj3X7ZsmRobG/WTn/xEycnJ7seiRYskSRaLRVu3btV1112nwYMH6xe/+IXGjh2rTz/9tNf6fM4nq1+0JKmQ02AAABjC8DlAnuhC5gh0xpubDuk3r3+lrH7R+scvJ3X78QEA8EdeMwfIX2X1b2mE/vqwXQ3N7V+aDwAAeg4ByADpcaGKDQtSo8Op7SV2o8sBAMDvEIAMYDKZ6AMCAMBABCCDtE6E3sxEaAAAeh0ByCBZ7ltisAIEAEBvIwAZZFRalCxmk47Y6nXEdsLocgAA8CsEIIOEBgVoaFKEJKnwYJWxxQAA4GcIQAZqPQ22iUZoAAB6FQHIQFn9oyXRBwQAQG8jABmodQVoe4lN9U0MRAQAoLcQgAzULzZUcWFBanK4tL3EZnQ5AAD4DQKQgUwmk3seEI3QAAD0HgKQwegDAgCg9xGADHb6QESXy2VwNQAA+AcCkMEyU1sGIpbZG1Riqze6HAAA/AIByGChQQEaltw6EJHTYAAA9AYCkAfgvmAAAPQuApAHOBWAqowtBAAAP0EA8gCtAWgHAxEBAOgVBCAPkBYbovjwloGIXx9mICIAAD2NAOQB2gxEpA8IAIAeRwDyEFlMhAYAoNcQgDxEVr9oSQxEBACgNxCAPERmarQCzCaVVzfocNUJo8sBAMCnEYA8REiQRcOSIyVxOTwAAD2NAORB3KfBmAgNAECPIgB5kKz+LY3Qm7kSDACAHkUA8iCtV4JtL7EzEBEAgB5EAPIgqTEh6hNhVbPTpW0MRAQAoMcQgDyIyWSiDwgAgF5AAPIw3BkeAICeRwDyMK2N0JsOVjEQEQCAHkIA8jAj+0YpwGxSRU2DDh1nICIAAD2BAORhggMtGpHSOhCR02AAAPQEApAHct8ZnkZoAAB6BAHIA7X2AXFLDAAAegYByAO1Xgq/84hdJxoZiAgAQHcjAHmgvtEhSjg5EHHroSqjywEAwOcQgDxQy0BEToMBANBTCEAeKqt/tCSuBAMAoCcQgDxU6wrQ5qLjDEQEAKCbEYA81MV9oxRoMamiplHFxxiICABAdyIAeajgQIuGp0RJ4jQYAADdjQDkwdx3hicAAQDQrQhAHow7wwMA0DMIQB6sdSL0ziPVqmtsNrgaAAB8BwHIg6VEBSsx0iqH06Wth2xGlwMAgM8gAHmwtgMROQ0GAEB3IQB5OHcAOlhlbCEAAPgQApCHa+0DYiAiAADdhwDk4S7uG6kgi1mVtY0qOlZndDkAAPgEApCHswZYNKJvpCT6gAAA6C4EIC9AHxAAAN2LAOQFuBIMAIDuRQDyAln9oyVJu0qrVdvAQEQAALqKAOQFkqNClBwVLIfTpa8OVRldDgAAXo8A5CVaT4NtLqoythAAAHwAAchLjGm9M/xB+oAAAOgqApCXcA9ELK5iICIAAF1EAPISI1JaBiIeq23Ut5UMRAQAoCsIQF7CGmDRxa0DETkNBgBAlxCAvAjzgAAA6B4eEYCWLl2q9PR0BQcHKzs7Wxs3bjzrvs8995wuu+wyxcTEKCYmRjk5OWfs73K5NG/ePCUnJyskJEQ5OTnau3dvT7+MHtfaB1TIlWAAAHSJ4QFo5cqVys3N1fz581VYWKhRo0ZpypQpKi8vb3f/NWvW6KabbtLHH3+sgoICpaWl6aqrrtLhw4fd+zzxxBN65plntHz5cn3++ecKCwvTlClTVF9f31svq0e0rgDtLrWrhoGIAAB0msll8CVF2dnZGj9+vJYsWSJJcjqdSktL0z333KM5c+ac9/kOh0MxMTFasmSJZsyYIZfLpZSUFP3mN7/Rb3/7W0mSzWZTYmKiXnzxRU2fPv28x7Tb7YqKipLNZlNkZGTXXmA3m5iXrxJbvV65LVsTB8UbXQ4AAB7jQj6/DV0Bamxs1KZNm5STk+PeZjablZOTo4KCgg4do66uTk1NTYqNjZUkHThwQKWlpW2OGRUVpezs7LMes6GhQXa7vc3DU43pTx8QAABdZWgAqqiokMPhUGJiYpvtiYmJKi0t7dAxHnzwQaWkpLgDT+vzLuSYeXl5ioqKcj/S0tIu9KX0mlON0FXGFgIAgBczvAeoKxYuXKhXX31Vb731loKDgzt9nLlz58pms7kfxcXF3Vhl98o6ORF6c9FxBiICANBJhgag+Ph4WSwWlZWVtdleVlampKSkcz530aJFWrhwoT744ANlZma6t7c+70KOabVaFRkZ2ebhqUakRCkowKzjdU06UFFrdDkAAHglQwNQUFCQxo4dq/z8fPc2p9Op/Px8TZgw4azPe+KJJ7RgwQKtXr1a48aNa/O7jIwMJSUltTmm3W7X559/fs5jeougALNG9o2SxGkwAAA6y/BTYLm5uXruuef00ksvaefOnbrzzjtVW1urWbNmSZJmzJihuXPnuvd//PHH9fDDD2vFihVKT09XaWmpSktLVVNTI0kymUy699579eijj+qdd97Rtm3bNGPGDKWkpOj666834iV2u9bTYDRCAwDQOQFGFzBt2jQdPXpU8+bNU2lpqUaPHq3Vq1e7m5iLiopkNp/KacuWLVNjY6N+8pOftDnO/Pnz9cgjj0iSHnjgAdXW1uqOO+5QVVWVJk+erNWrV3epT8iTtDRCH+CWGAAAdJLhc4A8kSfPAZKkMnu9sh/Ll9kkbX1kisKthudYAAAM5zVzgNA5iZHB6hsdIqdL+qq4yuhyAADwOgQgL+W+LxinwQAAuGAEIC9FIzQAAJ1HAPJSrROhNxdXMRARAIALRADyUsOSI2UNMKuqrknfMBARAIALQgDyUkEBZmWmnhyISB8QAAAXhADkxU7dGJUABADAhSAAebExrQHoYJWxhQAA4GUIQF4sq3+0JGlPebXs9U3GFgMAgBchAHmxhIhgpcaEyMVARAAALggByMtlcRoMAIALRgDycgxEBADgwhGAvFzrLTE2Fx2X08lARAAAOoIA5OWGJUcqONAse32zvqmoMbocAAC8AgHIywVazMrsGy2JPiAAADqKAOQDxpy8HJ4+IAAAOoYA5AOYCA0AwIUhAPmA1gC0t7yGgYgAAHQAAcgH9ImwKi22ZSDilqIqo8sBAMDjEYB8BKfBAADoOAKQjzgVgKqMLQQAAC9AAPIRrQGIgYgAAJwfAchHDE2OUHCgWdX1zdp/lIGIAACcCwHIRwRazBqVGi2JPiAAAM6HAORDWu8LxkRoAADOjQDkQ7gSDACAjiEA+ZAx/aIltQxEtJ1gICIAAGdDAPIh8eFW9Y8LlSRtKa4ythgAADwYAcjHuE+DHeQ0GAAAZ0MA8jFZJ0+D0QcEAMDZEYB8zJiTK0BbiqoYiAgAwFkQgHzM0KQIhQZZVN3QrL3lDEQEAKA9BCAfE2AxKzM1ShKnwQAAOBsCkA+iERoAgHMjAPkgBiICAHBuBCAf1DoQcf/RWlXVNRpbDAAAHogA5IPiwq1KPzkQcTMDEQEAOAMByEe1ngbbTB8QAABnIAD5qDGtd4YvqjK2EAAAPBAByEe1ToTeUlwlBwMRAQBogwDko4YktgxErGlo1t7yaqPLAQDAoxCAfFSAxaxRqdGSpMKDVYbWAgCApyEA+bCs/tGSmAcEAMC/IgD5MAYiAgDQPgKQD2u9M/w3DEQEAKANApAPiw0LUkZ8mCRpM5fDAwDgRgDyca23xeA0GAAApxCAfNzY/vQBAQDwrwhAPq61EXpLEQMRAQBoRQDycYMTIxRuDVBto0N7yhiICACARADyeRazSaPSoiRxGgwAgFYEID/gngfERGgAACQRgPxCawDazAoQAACSCEB+ofVS+G8qanWsloGIAAAQgPxAdGiQBvRpHYjIKhAAAAQgP8F9wQAAOIUA5CdohAYA4BQCkJ/I6h8tSfrqUJWaHU5jiwEAwGAEID9xUULLQMS6Rod2MxARAODnCEB+wmI2aXRatCSpkDvDAwD8HAHIj2SdvBx+80EaoQEA/o0A5EfGcGd4AAAkEYD8SlZaSwD6trJOlTUNBlcDAIBxDA9AS5cuVXp6uoKDg5Wdna2NGzeedd/t27frhhtuUHp6ukwmkxYvXnzGPo888ohMJlObx9ChQ3vwFXiPqNBADXQPRKwythgAAAxkaABauXKlcnNzNX/+fBUWFmrUqFGaMmWKysvL292/rq5OAwYM0MKFC5WUlHTW444YMUJHjhxxP9atW9dTL8HrMBARAACDA9DTTz+t22+/XbNmzdLw4cO1fPlyhYaGasWKFe3uP378eD355JOaPn26rFbrWY8bEBCgpKQk9yM+Pr6nXoLXyaIPCAAA4wJQY2OjNm3apJycnFPFmM3KyclRQUFBl469d+9epaSkaMCAAbr55ptVVFR0zv0bGhpkt9vbPHxV6wrQV8U2BiICAPyWYQGooqJCDodDiYmJbbYnJiaqtLS008fNzs7Wiy++qNWrV2vZsmU6cOCALrvsMlVXn334X15enqKiotyPtLS0Tv99T3dRQrgirAE60eTQrlIGIgIA/JPhTdDd7eqrr9ZPf/pTZWZmasqUKXr//fdVVVWl11577azPmTt3rmw2m/tRXFzcixX3LrPZpNGt84A4DQYA8FOGBaD4+HhZLBaVlZW12V5WVnbOBucLFR0drcGDB2vfvn1n3cdqtSoyMrLNw5eNcTdCVxlbCAAABjEsAAUFBWns2LHKz893b3M6ncrPz9eECRO67e/U1NRo//79Sk5O7rZjervWidA0QgMA/FWAkX88NzdXM2fO1Lhx43TJJZdo8eLFqq2t1axZsyRJM2bMUN++fZWXlyeppXF6x44d7u8PHz6sLVu2KDw8XIMGDZIk/fa3v9UPf/hD9e/fXyUlJZo/f74sFotuuukmY16kB2pdATpYWaeKmgbFh5/9ijoAAHyRoQFo2rRpOnr0qObNm6fS0lKNHj1aq1evdjdGFxUVyWw+tUhVUlKiMWPGuH9etGiRFi1apMsvv1xr1qyRJB06dEg33XSTKisr1adPH02ePFkbNmxQnz59evW1ebKokEBdlBCuveU12lxUpSuHJ57/SQAA+BCTy+VyGV2Ep7Hb7YqKipLNZvPZfqAH39iqlV8W687vDtSDU5mUDQDwfhfy+e1zV4GhY7L6R0uSCrkzPADADxGA/FTrQMSthxiICADwP50KQC+99JLee+89988PPPCAoqOjNXHiRB08eLDbikPPGdgnXJHBDEQEAPinTgWgxx57TCEhIZKkgoICLV26VE888YTi4+N13333dWuB6BktAxG5LxgAwD91KgAVFxe7LztftWqVbrjhBt1xxx3Ky8vTp59+2q0Foue0zgPaRB8QAMDPdCoAhYeHq7KyUpL0wQcf6Morr5QkBQcH68SJE91XHXpUFitAAAA/1ak5QFdeeaVuu+02jRkzRnv27NEPfvADSdL27duVnp7enfWhB43uFy2TSSo+dkJHqxvUJ4KBiAAA/9CpFaClS5dqwoQJOnr0qN58803FxcVJkjZt2sTEZS8SGdwyEFFiFQgA4F86tQIUHR2tJUuWnLH9D3/4Q5cLQu/K6hejPWU1Kiw6rikjuu8mtAAAeLJOrQCtXr1a69atc/+8dOlSjR49Wj/72c90/DgrCd6ktQ9o88EqYwsBAKAXdSoA3X///bLb7ZKkbdu26Te/+Y1+8IMf6MCBA8rNze3WAtGzWidCbz1cpSYGIgIA/ESnToEdOHBAw4cPlyS9+eabuvbaa/XYY4+psLDQ3RAN7zAgvmUgor2+WTuP2JWZGm10SQAA9LhOrQAFBQWprq5OkvTRRx/pqquukiTFxsa6V4bgHcxmk8a0Xg7PPCAAgJ/oVACaPHmycnNztWDBAm3cuFHXXHONJGnPnj1KTU3t1gLR807NA6oythAAAHpJpwLQkiVLFBAQoDfeeEPLli1T3759JUn//Oc/NXXq1G4tED3PfWd4LoUHAPiJTvUA9evXT+++++4Z2//jP/6jywWh941OaxmIeOj4CZVX1yshItjokgAA6FGdCkCS5HA4tGrVKu3cuVOSNGLECF133XWyWCzdVhx6R0RwoAYnRGh3WbUKD1Zp6sXMAwIA+LZOnQLbt2+fhg0bphkzZugf//iH/vGPf+iWW27RiBEjtH///u6uEb2g9TTYZk6DAQD8QKcC0K9+9SsNHDhQxcXFKiwsVGFhoYqKipSRkaFf/epX3V0jesEYbowKAPAjnToFtnbtWm3YsEGxsbHubXFxcVq4cKEmTZrUbcWh97ReCbb1kE2NzU4FBXQqGwMA4BU69SlntVpVXV19xvaamhoFBQV1uSj0vgHxYYoODVRDs1M7jzDLCQDg2zoVgK699lrdcccd+vzzz+VyueRyubRhwwb9+7//u6677rrurhG9wGw2aUxatCROgwEAfF+nAtAzzzyjgQMHasKECQoODlZwcLAmTpyoQYMGafHixd1cInoLAxEBAP6iUz1A0dHRevvtt7Vv3z73ZfDDhg3ToEGDurU49K6s/twSAwDgHzocgM53l/ePP/7Y/f3TTz/d+YpgmFFp0TKbpMNVJ1Rur1dCJAMRAQC+qcMBaPPmzR3az2QydboYGCvcGqDBiRHaVVqtwqLjmnpxstElAQDQIzocgE5f4YHvyuofczIAVRGAAAA+i2EvaMPdCE0fEADAhxGA0EZWv2hJ0tbDLQMRAQDwRQQgtJERH6aY0EA1Nju1vcRmdDkAAPQIAhDaMJlMp90XrMrYYgAA6CEEIJyh9TQYE6EBAL6KAIQztDZCb6YRGgDgowhAOEPrQMQSW71KbfVGlwMAQLcjAOEMYdYADUmKlMRpMACAbyIAoV3uPiBOgwEAfBABCO06dWd4AhAAwPcQgNCu1jvDf33YroZmh8HVAADQvQhAaFd6XKhiw4LU6HBqe4nd6HIAAOhWBCC0y2QyaUxatCT6gAAAvocAhLNqPQ22mYnQAAAfQwDCWY1hIjQAwEcRgHBWo1JbBiIesdXriO2E0eUAANBtCEA4qzBrgIa2DkQ8WGVsMQAAdCMCEM5pbH/mAQEAfA8BCOeU1T9aEgEIAOBbCEA4p9aJ0NsZiAgA8CEEIJxTv9hQxZ0ciPj1YQYiAgB8AwEI52QymTSmX+s8IE6DAQB8AwEI50UfEADA1xCAcF7uO8NzKTwAwEcQgHBemalRsphNKrXXq6SKgYgAAO9HAMJ5hQYFaFhyhCRpEzdGBQD4AAIQOsR9Gow+IACADyAAoUNOBaAqYwsBAKAbEIDQIa0BaEeJTfVNDEQEAHg3AhA6JC02RPHhQWpyuPT1YZvR5QAA0CUEIHTI6QMR6QMCAHg7AhA6jHlAAABfQQBCh2X1i5bUsgLkcrmMLQYAgC4gAKHDMlOjFWA2qby6QYcZiAgA8GIEIHRYSJBFw5IjJXE5PADAuxGAcEHcp8GYCA0A8GKGB6ClS5cqPT1dwcHBys7O1saNG8+67/bt23XDDTcoPT1dJpNJixcv7vIxcWGy+rc0Qm/mSjAAgBczNACtXLlSubm5mj9/vgoLCzVq1ChNmTJF5eXl7e5fV1enAQMGaOHChUpKSuqWY+LCtF4Jtr3EzkBEAIDXMjQAPf3007r99ts1a9YsDR8+XMuXL1doaKhWrFjR7v7jx4/Xk08+qenTp8tqtXbLMXFhUmNCFB9uVbPTpW0MRAQAeCnDAlBjY6M2bdqknJycU8WYzcrJyVFBQUGvHrOhoUF2u73NA+0zmUz0AQEAvJ5hAaiiokIOh0OJiYltticmJqq0tLRXj5mXl6eoqCj3Iy0trVN/31+MPdkH9MrGIn1bUWtwNQAAXDjDm6A9wdy5c2Wz2dyP4uJio0vyaP82pq+SIoN1sLJOP1q6Xp/tqzC6JAAALohhASg+Pl4Wi0VlZWVttpeVlZ21wbmnjmm1WhUZGdnmgbNLiAzWO3dP0qi0aNlONOnnKzbqrxsOGl0WAAAdZlgACgoK0tixY5Wfn+/e5nQ6lZ+frwkTJnjMMdG+hMhgrbzjUl0/OkUOp0sPr/paD6/6Wk0Op9GlAQBwXgFG/vHc3FzNnDlT48aN0yWXXKLFixertrZWs2bNkiTNmDFDffv2VV5enqSWJucdO3a4vz98+LC2bNmi8PBwDRo0qEPHRPcJDrToP6aN1kWJEXryf3frrxsOav/RGj17c5aiQ4OMLg8AgLMyNABNmzZNR48e1bx581RaWqrRo0dr9erV7ibmoqIimc2nFqlKSko0ZswY98+LFi3SokWLdPnll2vNmjUdOia6l8lk0l3fG6SLEsJ178ot+mx/pa5ful7/NXOcBiVEGF0eAADtMrm4rfcZ7Ha7oqKiZLPZ6Ae6ADuP2HXbS1/qcNUJRVgD9J8/G6PvDkkwuiwAgJ+4kM9vrgJDtxmWHKm3756k8ekxqm5o1uwXv9Dz6w6IjA0A8DQEIHSr+HCr/nZbtn46NlVOl7Tg3R2a8+Y2NTbTHA0A8BwEIHQ7a4BFT/wkU7+/ZpjMJmnll8W65b8+V2VNg9GlAQAgiQCEHmIymXTbZQP0/K3jFWEN0MZvj+m6Jeu1q5TbjAAAjEcAQo/63pAEvXXXRPWPC9XhqhO64dnP9OGOsvM/EQCAHkQAQo8blBChVb+cpIkD41Tb6NAdf/1Sz67ZR3M0AMAwBCD0ipiwIL00+xL9/NL+crmkJ1bv1n0rt6i+yWF0aQAAP0QAQq8JtJi14PqLteBHI2Qxm7RqS4mm/2WDyu31RpcGAPAzBCD0up9PSNdfZ1+iqJBAbSmu0o+WrtfXh21GlwUA8CMEIBhi4qB4vX3XJA3sE6Yjtnr9ZPlnem/rEaPLAgD4CQIQDJMeH6a37pqkywf3UX2TU3e9UqjFH+2R00lzNACgZxGAYKjI4ECtuHW8bpucIUla/NFe3f33QtU1NhtcGQDAlxGAYDiL2aTfXztcT9yQqUCLSe9vK9VPlxeopOqE0aUBAHwUAQge48bxaXrl9ksVGxak7SV2XbdkvQqLjhtdFgDABxGA4FHGp8fq7bsmaWhShCpqGjT9Lxv0j8JDRpcFAPAxBCB4nLTYUL1x50TlDEtUY7NTua99pYX/3EVzNACg2xCA4JHCrQH6y8/H6pffHShJWr52v+7465eqaaA5GgDQdQQgeCyz2aQHpg7V4mmjFRRg1kc7y3XDs5+p+Fid0aUBALwcAQge7/oxfbXyjkvVJ8Kq3WXV+tHS9fr8m0qjywIAeDECELzCmH4xeufuSbq4b6SO1Tbqluc/18oviowuCwDgpQhA8BrJUSF6/f9N1DUjk9XkcOnBN7fpj/+zQ80Op9GlAQC8DAEIXiUkyKIlPxuj+3IGS5JWrD+g2S99KduJJoMrAwB4EwIQvI7JZNKvcy7SszdnKTjQrE/2HNW/PbteBypqjS4NAOAlCEDwWj8Ymaw3/n2ikqOC9c3RWl2/dL3W7a0wuiwAgBcgAMGrXdw3Sm/fPUlj+kXLdqJJM1/YqJcLvjW6LACAhyMAweslRATr77dfqh+P6SuH06V5b2/X71dtUxPN0QCAsyAAwScEB1r01I2jNOfqoTKZpL9tKNKM5zfqeG2j0aUBADwQAQg+w2Qy6d8vH6jnfj5OYUEWFXxTqeufXa995dVGlwYA8DAEIPicnOGJ+scvJyk1JkQHK+v0b0s/08e7y40uCwDgQQhA8ElDkiL09l2TdElGrKobmvWLF7/Qf336jVwu7igPACAAwYfFhVv1t19ka/r4NDld0qPv7dQDb2xVQ7PD6NIAAAYjAMGnBQWYlffjkZp37XCZTdLrmw7p5uc+V0VNg9GlAQAMRACCzzOZTJo9OUMvzLpEEcEB+vLgcf1oyXrtPGI3ujQAgEEIQPAblw/uo7d+OUnpcaE6XHVCNyz7TP+7vdTosgAABiAAwa8MSgjXqrsmadKgONU1OvT//rpJj7yzXYeO1xldGgCgF5lcXBZzBrvdrqioKNlsNkVGRhpdDnpAk8OpR9/doZcKDkqSLGaTpo5I0uzJGRrbP8bg6gAAnXEhn98EoHYQgPzHmt3leu7Tb7R+X6V72+i0aP1icoauvjhJARYWSQHAWxCAuogA5H92HrFrxboDentLiRpP3kMsJSpYMyema/ol/RQVEmhwhQCA8yEAdREByH8drW7Q3zYc1N82HFTlyfuIhQZZ9JOxqZo1KUMZ8WEGVwgAOBsCUBcRgFDf5NA7W0q0Yv0B7SptuZeYySRdMTRBsydnaMKAOJlMJoOrBACcjgDURQQgtHK5XPpsf6WeX3dA/7fr1P3EhidHavbkDP1wVLKsARYDKwQAtCIAdREBCO3Zf7RGL6w/oDc2HVJ9U0ufUJ8Iq35+aX/dnN1PceFWgysEAP9GAOoiAhDOpaquUa9sLNLLnx1Uqb1eUsstN348pq9mT87Q4MQIgysEAP9EAOoiAhA6osnh1PvbjmjFugP66pDNvf2yi+I1e3KGLr+oj8xm+oQAoLcQgLqIAIQL4XK5tOngcT2/7oD+d3upnCf/RQ3sE6bZkzP04zGpCgmiTwgAehoBqIsIQOis4mN1evGzb7Xyi2LVNDRLkqJDA3Vzdj/NmJCuxMhggysEAN9FAOoiAhC6qrq+Sa9/eUgvfHZAxcdOSJICLSZdm5mi2ZMyNDI1yuAKAcD3EIC6iACE7uJwuvThjjKtWHdAG7895t5+SXqsZk/O0JXDE2WhTwgAugUBqIsIQOgJ2w7Z9Py6b/Tu1iNqPtko1C82VLdOTNeN49MUbg0wuEIA8G4EoC4iAKEnldrq9XLBt3plY5Gq6pokSRHWAE0bn6aZE9OVFhtqcIUA4J0IQF1EAEJvONHo0D82H9KKdQe0/2itJMlskqaMSNIvJmdobP8YbrcBABeAANRFBCD0JqfTpbV7j2rFugP6dG+Fe/uo1CjNnpyhH4xMVqDFbGCFAOAdCEBdRACCUXaXVmvFugN6a8thNTa33G4jOSpYMyak62eX9FNUaKDBFQKA5yIAdREBCEarqGnQK58X6eWCg6qoaZAkhQRa9JOxqZo1KV0D+oQbXCEAeB4CUBcRgOApGpod+p+vjuj5dQe084jdvf37QxP0i8kZmjgwjj4hADiJANRFBCB4GpfLpYJvKrVi3QHl7ypX67/aoUkRmj05Q9eNSlFwILfbAODfCEBdRACCJztQUasX1h/Q618e0okmhyQpPjxIP85K1XeH9NG4/rEKCqBpGoD/IQB1EQEI3sBW16RXvyjSS599qxJbvXt7uDVAEwfG6XtDE/TdIX2UHBViYJUA0HsIQF1EAII3aXI49dGOMn24o0xr9xxVZW1jm98PSYzQd4f00eWsDgHwcQSgLiIAwVs5nS59XWLTmt1HtWZ3ubYUV8l52r/wsCCLJg2K13eHtKwOpUSzOgTAdxCAuogABF9xvLZRn+w9qrW7j7I6BMDnEYC6iAAEX8TqEABfRwDqIgIQ/MHx2kZ9uq9Ca3aXa+3uM1eHBieGu8MQq0MAvMGFfH57xH/Rli5dqvT0dAUHBys7O1sbN2485/6vv/66hg4dquDgYI0cOVLvv/9+m9/feuutMplMbR5Tp07tyZcAeJ2YsCBdNypFT984Wl/8Lkfv3D1JuVcOVla/aJlN0p6yGv3lk2/0s+c+15g/fqA7Xv5Sr3xepJKqE0aXDgBdZvgK0MqVKzVjxgwtX75c2dnZWrx4sV5//XXt3r1bCQkJZ+z/2Wef6Tvf+Y7y8vJ07bXX6pVXXtHjjz+uwsJCXXzxxZJaAlBZWZleeOEF9/OsVqtiYmI6VBMrQPB3rA4B8EZedQosOztb48eP15IlSyRJTqdTaWlpuueeezRnzpwz9p82bZpqa2v17rvvurddeumlGj16tJYvXy6pJQBVVVVp1apVHaqhoaFBDQ0N7p/tdrvS0tIIQIDoHQLgPS4kAAX0Uk3tamxs1KZNmzR37lz3NrPZrJycHBUUFLT7nIKCAuXm5rbZNmXKlDPCzpo1a5SQkKCYmBh9//vf16OPPqq4uLh2j5mXl6c//OEPXXsxgI8ym03KTI1WZmq0fnXFRe2uDn2wo0wf7CiTdNrq0OA+GpfO6hAAz2RoAKqoqJDD4VBiYmKb7YmJidq1a1e7zyktLW13/9LSUvfPU6dO1Y9//GNlZGRo//79euihh3T11VeroKBAFsuZ90uaO3dum1DVugIE4EytvUPXjUppd3VoT1mNu3+I1SEAnsrQANRTpk+f7v5+5MiRyszM1MCBA7VmzRpdccUVZ+xvtVpltVp7s0TAJ7A6BMBbGRqA4uPjZbFYVFZW1mZ7WVmZkpKS2n1OUlLSBe0vSQMGDFB8fLz27dvXbgAC0D1YHQLgLQwNQEFBQRo7dqzy8/N1/fXXS2ppgs7Pz9fdd9/d7nMmTJig/Px83Xvvve5tH374oSZMmHDWv3Po0CFVVlYqOTm5O8sHcA7/ujpUVdeoT/a2rA59sueoKmraXx2aMCBOw5IjlRhplclkMvhVAPBVhl8FtnLlSs2cOVN//vOfdckll2jx4sV67bXXtGvXLiUmJmrGjBnq27ev8vLyJLVcBn/55Zdr4cKFuuaaa/Tqq6/qsccec18GX1NToz/84Q+64YYblJSUpP379+uBBx5QdXW1tm3b1qFTXVwGD/Qsp9Ol7SV2fby7vN0ryyQpJjRQw5IjT3tEaFBCuKwBZ/bxAYDkRVeBSS2XtR89elTz5s1TaWmpRo8erdWrV7sbnYuKimQ2n+oTmDhxol555RX9/ve/10MPPaSLLrpIq1atcs8Aslgs2rp1q1566SVVVVUpJSVFV111lRYsWECfD+AhzGaTRqZGaWRqVJvVobW7j+qrQ1X65miNjtc16bP9lfpsf6X7eQFmkwb2Cdew5Ig24ahPBP+2AVwYw1eAPBErQICx6psc2lNWrZ1H7Np5pPWrXfb65nb3jw+3nhaKWr4O7BOuQAtN1oA/8apBiJ6IAAR4HpfLpRJbvXaWtIShnaV27TpSrQOVtWrvv2JBFrMGJYS7Q9Hw5EgNTY5UbFhQ7xcPoFcQgLqIAAR4j7rGZu0urW6zUrSrtFo1De2vFiVGWtucPhueHKH0uDAFsFoEeD0CUBcRgADv5nK5dOj4Ce04GYhaT6UVHatrd39rgFmDEyPa9hYlRSoqNLCXKwfQFQSgLiIAAb6ppqFZu0vt2nHaatHu0mrVNTra3b9vdIiGJUdoaNKpK9H6x4XJYubyfMATEYC6iAAE+A+n06WiY3XuQNQajg5XnWh3/5BAi4YkRbhPnw1NjtTQpAhFBLNaBBiNANRFBCAAthNNJ3uLTp1G211WrfomZ7v7p8WGaFjSqd6iESmRSo0JYZgj0IsIQF1EAALQHofTpQMVtdpVam9zif4RW327+0eHBmpk3yhlpkZpZN9oZaZGKTkqmFAE9BACUBcRgABciOO1jdp1+mpRaUtvUZPjzP+8xocHaWTfKI1MjVbmyXCUEBlsQNWA7yEAdREBCEBXNTQ7tKe0RlsPV2nbIZu2HrJpT1m1mv/1nh9quTS/dYVoZGqURvaNUnw4062BC0UA6iICEICeUN/k0M4jdm07bNO2QzZtO9wSitrJROobHaKL+0YqMzW6ZcWob5RiGOIInBMBqIsIQAB6S11js3YesWvroZZQtPWwTfuP1rQ73TotNkSZfaM1MjVKmX2jNKJvlKJCuPoMaEUA6iICEAAj1TQ0a/vhlhWirSdXig5U1La7b0Z82GmN1i2hKNxq+H2uAUMQgLqIAATA09hONGn74ZYVopaVoioVHztzVpHJJA3sE67Mvi39RJmpURqeHKWQIIsBVQO9iwDURQQgAN7geG1jSz/RYZu2Hmppti5p55J8s0kanBhxaqUoNVpDkyIUHEgogm8hAHURAQiAtzpa3aCv3afOqvTVIZuOVjecsV+A2aTBiRHuK88y+0ZrSFKEggK4KSy8FwGoiwhAAHxJmb3+ZJN1lfsUWmVt4xn7BVnMGpoc0WZ440WJ4Qq0EIrgHQhAXUQAAuDLXC6XSmz1LYHo0Klma9uJpjP2DbSYlBYTqvT4MKXHhSk9PlTpcWHKiA9TSnQIN4aFRyEAdREBCIC/cblcKj52os3gxq8P21Td0HzW5wRaTEqLDVVGXJj6x4UpI/5UUCIcwQgEoC4iAAGA5HS6dMRer28ranWgolYHK2t1oKJO31bWqqiyTo2O9m8MK7WcTkuLDTm5anTyERdKOEKPIgB1EQEIAM7N4XTpiO2Evj0ZiL6tqG35WlnX4XCUcXK1qH98mDJOnl5LjiIcofMIQF1EAAKAzjs9HB04GY5aVo9qVXzsxLnDUYBZ/WJbVorS41pOqWXEh6l/XKhSokJkJhzhHAhAXUQAAoCe4XC6VFJ1wr1a9G3FqdWjomN1anKc/SMpKMCs/rGhZ/QbpceHKTkymHAEAlBXEYAAoPe1CUcVLf1GBytrdaCyVsUdDEenrxhlnAxHSYQjv0EA6iICEAB4lmaHU0ds9TrQ2mt0Wu9R0bE6NTvP/lFmDTArJTpEwYEWBQWYZbWYZQ00K8hibvk5oPVry+/b23b6c1q+Wtrs96/PsQa07Evw6l0X8vnNHfMAAB4vwGJWWmyo0mJD9R31afO7ZodTJVX1OlB5qteope+oTkXH6tTQ7DzrzWR7WqDFdDI0WdoJXK3hyXJqW5ugdeZzQgItSoi0KjEyWEmRwYoNC5LJRMjqDAIQAMCrBVjM6hcXqn5xoVI74ehw1QmV2urV0OxUY7Oz5avDoYYmpxodp7a1PBxqPH2/07c5nGc8p/X3px/7dE0Ol5ocDtU2OnrktQdZzEqItCopMliJUS2h6F+/T4i0ct+3dhCAAAA+K8BiVv+Tgxp7g8vlUpPD1W5oavnqcIet00NUY3vhy+FUQ5Oj5evJbXUNzSqzN6jMXq/K2kY1Opw6dPyEDh0/cc66YkIDW1aNTgaj9r6PCQ30q9UkAhAAAN3EZDIpKMDUKzeVbWh2qPxkGCq116vUVn/y+waV2U5us9ersdmp43VNOl7XpF2l1Wc9XlCAWYmtq0knV4+SotqGpYRIq6wBvrGaRAACAMALWQMs7r6os3G5XKqqa3KHodZg1LqK1BqaKmsb1djsVPGxEyo+du7VpNiwoJMByXoqIP3LabdoL1hNIgABAOCjTCaTYsKCFBMWpGHJZ78q6kJWk47VNupYbaN2Hjn737UGmP8lGFnPOO2WGBncKytlZ0MAAgDAz3VtNaklMJWeDFDHahvV0OxU0bGWq/DOZuaE/vrDjy7uiZfTIQQgAABwXhe6mtRmJen0sHTyFFxiVHAvVn8mAhAAAOg2HV1NOtfwyt5AAAIAAL3KZDIp0GJsk7Rx3UcAAAAGIQABAAC/QwACAAB+hwAEAAD8DgEIAAD4HQIQAADwOwQgAADgdwhAAADA7xCAAACA3yEAAQAAv0MAAgAAfocABAAA/A4BCAAA+B3uBt8Ol8slSbLb7QZXAgAAOqr1c7v1c/xcCEDtqK6uliSlpaUZXAkAALhQ1dXVioqKOuc+JldHYpKfcTqdKikpUUREhEwmU7ce2263Ky0tTcXFxYqMjOzWY+PC8X54Ft4Pz8L74Vl4P87P5XKpurpaKSkpMpvP3eXDClA7zGazUlNTe/RvREZG8n/AHoT3w7PwfngW3g/Pwvtxbudb+WlFEzQAAPA7BCAAAOB3CEC9zGq1av78+bJarUaXAvF+eBreD8/C++FZeD+6F03QAADA77ACBAAA/A4BCAAA+B0CEAAA8DsEIAAA4HcIQL1o6dKlSk9PV3BwsLKzs7Vx40ajS/JLeXl5Gj9+vCIiIpSQkKDrr79eu3fvNrosnLRw4UKZTCbde++9Rpfi1w4fPqxbbrlFcXFxCgkJ0ciRI/Xll18aXZZfcjgcevjhh5WRkaGQkBANHDhQCxYs6ND9rnB2BKBesnLlSuXm5mr+/PkqLCzUqFGjNGXKFJWXlxtdmt9Zu3at7rrrLm3YsEEffvihmpqadNVVV6m2ttbo0vzeF198oT//+c/KzMw0uhS/dvz4cU2aNEmBgYH65z//qR07duipp55STEyM0aX5pccff1zLli3TkiVLtHPnTj3++ON64okn9J//+Z9Gl+bVuAy+l2RnZ2v8+PFasmSJpJb7jaWlpemee+7RnDlzDK7Ovx09elQJCQlau3atvvOd7xhdjt+qqalRVlaWnn32WT366KMaPXq0Fi9ebHRZfmnOnDlav369Pv30U6NLgaRrr71WiYmJev75593bbrjhBoWEhOhvf/ubgZV5N1aAekFjY6M2bdqknJwc9zaz2aycnBwVFBQYWBkkyWazSZJiY2MNrsS/3XXXXbrmmmva/DuBMd555x2NGzdOP/3pT5WQkKAxY8boueeeM7osvzVx4kTl5+drz549kqSvvvpK69at09VXX21wZd6Nm6H2goqKCjkcDiUmJrbZnpiYqF27dhlUFaSWlbh7771XkyZN0sUXX2x0OX7r1VdfVWFhob744gujS4Gkb775RsuWLVNubq4eeughffHFF/rVr36loKAgzZw50+jy/M6cOXNkt9s1dOhQWSwWORwO/elPf9LNN99sdGlejQAEv3bXXXfp66+/1rp164wuxW8VFxfr17/+tT788EMFBwcbXQ7U8v8YjBs3To899pgkacyYMfr666+1fPlyApABXnvtNf33f/+3XnnlFY0YMUJbtmzRvffeq5SUFN6PLiAA9YL4+HhZLBaVlZW12V5WVqakpCSDqsLdd9+td999V5988olSU1ONLsdvbdq0SeXl5crKynJvczgc+uSTT7RkyRI1NDTIYrEYWKH/SU5O1vDhw9tsGzZsmN58802DKvJv999/v+bMmaPp06dLkkaOHKmDBw8qLy+PANQF9AD1gqCgII0dO1b5+fnubU6nU/n5+ZowYYKBlfknl8ulu+++W2+99Zb+7//+TxkZGUaX5NeuuOIKbdu2TVu2bHE/xo0bp5tvvllbtmwh/Bhg0qRJZ4yG2LNnj/r3729QRf6trq5OZnPbj2uLxSKn02lQRb6BFaBekpubq5kzZ2rcuHG65JJLtHjxYtXW1mrWrFlGl+Z37rrrLr3yyit6++23FRERodLSUklSVFSUQkJCDK7O/0RERJzRfxUWFqa4uDj6sgxy3333aeLEiXrsscd04403auPGjfrLX/6iv/zlL0aX5pd++MMf6k9/+pP69eunESNGaPPmzXr66ac1e/Zso0vzalwG34uWLFmiJ598UqWlpRo9erSeeeYZZWdnG12W3zGZTO1uf+GFF3Trrbf2bjFo13e/+10ugzfYu+++q7lz52rv3r3KyMhQbm6ubr/9dqPL8kvV1dV6+OGH9dZbb6m8vFwpKSm66aabNG/ePAUFBRldntciAAEAAL9DDxAAAPA7BCAAAOB3CEAAAMDvEIAAAIDfIQABAAC/QwACAAB+hwAEAAD8DgEIAAD4HQIQAHTAmjVrZDKZVFVVZXQpALoBAQgAAPgdAhAAAPA7BCAAXsHpdCovL08ZGRkKCQnRqFGj9MYbb0g6dXrqvffeU2ZmpoKDg3XppZfq66+/bnOMN998UyNGjJDValV6erqeeuqpNr9vaGjQgw8+qLS0NFmtVg0aNEjPP/98m302bdqkcePGKTQ0VBMnTtTu3bt79oUD6BEEIABeIS8vTy+//LKWL1+u7du367777tMtt9yitWvXuve5//779dRTT+mLL75Qnz599MMf/lBNTU2SWoLLjTfeqOnTp2vbtm165JFH9PDDD+vFF190P3/GjBn6+9//rmeeeUY7d+7Un//8Z4WHh7ep43e/+52eeuopffnllwoICNDs2bN75fUD6F7cDR6Ax2toaFBsbKw++ugjTZgwwb39tttuU11dne644w5973vf06uvvqpp06ZJko4dO6bU1FS9+OKLuvHGG3XzzTfr6NGj+uCDD9zPf+CBB/Tee+9p+/bt2rNnj4YMGaIPP/xQOTk5Z9SwZs0afe9739NHH32kK664QpL0/vvv65prrtGJEycUHBzcw/8rAOhOrAAB8Hj79u1TXV2drrzySoWHh7sfL7/8svbv3+/e7/RwFBsbqyFDhmjnzp2SpJ07d2rSpEltjjtp0iTt3btXDodDW7ZskcVi0eWXX37OWjIzM93fJycnS5LKy8u7/BoB9K4AowsAgPOpqamRJL333nvq27dvm99ZrdY2IaizQkJCOrRfYGCg+3uTySSppT8JgHdhBQiAxxs+fLisVquKioo0aNCgNo+0tDT3fhs2bHB/f/z4ce3Zs0fDhg2TJA0bNkzr169vc9z169dr8ODBslgsGjlypJxOZ5ueIgC+ixUgAB4vIiJCv/3tb3XffffJ6XRq8uTJstlsWr9+vSIjI9W/f39J0h//+EfFxcUpMTFRv/vd7xQfH6/rr79ekvSb3/xG48eP14IFCzRt2jQVFBRoyZIlevbZZyVJ6enpmjlzpmbPnq1nnnlGo0aN0sGDB1VeXq4bb7zRqJcOoIcQgAB4hQULFqhPnz7Ky8vTN998o+joaGVlZemhhx5yn4JauHChfv3rX2vv3r0aPXq0/ud//kdBQUGSpKysLL322muaN2+eFixYoOTkZP3xj3/Urbfe6v4by5Yt00MPPaRf/vKXqqysVL9+/fTQQw8Z8XIB9DCuAgPg9Vqv0Dp+/Liio6ONLgeAF6AHCAAA+B0CEAAA8DucAgMAAH6HFSAAAOB3CEAAAMDvEIAAAIDfIQABAAC/QwACAAB+hwAEAAD8DgEIAAD4HQIQAADwO/8fFA8FivgQoKAAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 开始训练\n",
    "total_losses = []\n",
    "\n",
    "for epoch_ in range(epoch):\n",
    "    batch_losses = []\n",
    "    for x, y in train_loader:\n",
    "        x = x.to('cuda')\n",
    "        y = y.to('cuda')\n",
    "        pred = net(x)\n",
    "        loss = loss_fn(pred, y)\n",
    "        batch_losses.append(loss.item())\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "    total_losses.append(sum(batch_losses) / len(batch_losses))\n",
    "\n",
    "Fig = plt.figure()\n",
    "plt.plot(range(epoch), total_losses)\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "dac2eaf4c976c5a1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-07-26T08:37:04.124749Z",
     "start_time": "2025-07-26T08:37:04.004838Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正确率：97.88%\n"
     ]
    }
   ],
   "source": [
    "# 测试模型\n",
    "total = 0\n",
    "correct = 0\n",
    "with torch.no_grad():\n",
    "    for x, y in test_loader:\n",
    "        x = x.to('cuda')\n",
    "        y = y.to('cuda')\n",
    "        pred = net(x)\n",
    "        correct += torch.sum(torch.argmax(pred, dim=1) == y)\n",
    "        total += y.size(0)\n",
    "\n",
    "print(f\"正确率：{correct / total * 100:.2f}%\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "537759c2648dd476",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-07-26T08:37:04.219140Z",
     "start_time": "2025-07-26T08:37:04.159374Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8a9dd8e0863047cda62f3c8d72ad90c7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(FileUpload(value=(), accept='.png,.jpg,.jpeg', description='Upload'), HBox(children=(IntSlider(…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from tools.ipynb_draw import DrawCanvas\n",
    "from IPython.display import display\n",
    "from ipywidgets import Button\n",
    "from PIL import Image\n",
    "\n",
    "dc = DrawCanvas(width=58, height=58)\n",
    "\n",
    "callback_button = Button(description=\"识别\")\n",
    "\n",
    "def callback(_):\n",
    "    with dc.out:\n",
    "        try:\n",
    "            np_img = dc.canvas.get_image_data(width=58, height=58)\n",
    "            img = Image.fromarray(np_img)\n",
    "            transform = transforms.Compose([\n",
    "                transforms.Grayscale(),  # 如果是彩色图像转为灰度\n",
    "                transforms.Resize((28, 28)),  # 调整大小为28x28\n",
    "                transforms.ToTensor(),  # 转为张量并归一化到[0,1]\n",
    "                transforms.Normalize((0.1307,), (0.3081,))  # 标准化到[-1,1]\n",
    "            ])\n",
    "\n",
    "            # 对图像进行预处理\n",
    "            input_tensor = transform(img).unsqueeze(0).to('cuda')  # 添加batch维度 (1, 1, 28, 28)\n",
    "\n",
    "            # 3. 前向传播\n",
    "            with torch.no_grad():  # 不需要计算梯度\n",
    "                output = net(input_tensor)\n",
    "                prediction = torch.argmax(output, dim=1).item()  # 获取预测类别\n",
    "                print(prediction)\n",
    "        except BaseException as e:\n",
    "            print(str(e))\n",
    "\n",
    "callback_button.on_click(callback)\n",
    "\n",
    "\n",
    "layout = dc.get_layout()\n",
    "layout.children[1].children = layout.children[1].children + (callback_button,)\n",
    "display(layout)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1861f1d1-c468-4db2-bdf7-fd05f326e89e",
   "metadata": {},
   "source": [
    "实际上对于纯神经网络输入的识别效果并不好，必须是很接近训练集的数据才能准确的识别，对不同类型的图片的泛化能力其实不太行。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "825857ffea1cda6d",
   "metadata": {},
   "source": [
    "# 损失函数说明\n",
    "\n",
    "CrossEntropyLoss（交叉熵损失）、MSELoss（均方误差损失）和BCELoss（二元交叉熵损失）是深度学习中常用的损失函数，它们在应用场景、数学形式和适用任务上有显著差异。以下是它们的异同点分析：\n",
    "\n",
    "---\n",
    "\n",
    "### **1. 核心区别**\n",
    "| **损失函数**       | **主要应用场景**               | **输出类型**       | **数学形式**                          | **特点**                                                                 |\n",
    "|--------------------|-------------------------------|--------------------|---------------------------------------|--------------------------------------------------------------------------|\n",
    "| **CrossEntropyLoss** | 多分类任务（单标签）           | 概率分布（Softmax）| \\(-\\sum_{c=1}^C y_c \\log(p_c)\\)       | 结合Softmax，直接优化多类概率分布，对错误分类敏感。                       |\n",
    "| **BCELoss**         | 二分类任务或多标签分类         | 独立概率（Sigmoid）| \\(-\\frac{1}{N}\\sum_{i=1}^N [y_i \\log(p_i) + (1-y_i)\\log(1-p_i)]\\) | 每个输出节点独立计算二元概率，需显式Sigmoid激活。                         |\n",
    "| **MSELoss**         | 回归任务（连续值预测）         | 任意实数           | \\(\\frac{1}{N}\\sum_{i=1}^N (y_i - p_i)^2\\) | 对异常值敏感，假设误差服从高斯分布，分类任务中可能导致梯度消失。           |\n",
    "\n",
    "---\n",
    "\n",
    "### **2. 相同点**\n",
    "- **目标**：均用于衡量模型预测值与真实值的差异，指导模型参数优化。\n",
    "- **可微性**：均为可微函数，支持梯度下降法优化。\n",
    "\n",
    "---\n",
    "\n",
    "### **3. 关键差异**\n",
    "#### **(1) 任务类型**\n",
    "- **CrossEntropyLoss**：专为**单标签多分类**设计（如MNIST手写数字识别）。\n",
    "- **BCELoss**：适用于**二分类**或**多标签分类**（如同时预测多个疾病的概率）。\n",
    "- **MSELoss**：主要用于**回归问题**（如房价预测），分类任务中效果较差。\n",
    "\n",
    "#### **(2) 概率建模**\n",
    "- **CrossEntropyLoss**：隐含Softmax，强制输出为概率分布（\\(\\sum p_c = 1\\)）。\n",
    "- **BCELoss**：对每个输出节点单独应用Sigmoid，概率独立（适合非互斥类别）。\n",
    "- **MSELoss**：无概率约束，直接拟合数值。\n",
    "\n",
    "#### **(3) 梯度特性**\n",
    "- **CrossEntropyLoss**：梯度与误差成比例（\\(p_c - y_c\\)），错误分类时梯度大，收敛快。\n",
    "- **BCELoss**：类似但对每个节点独立计算。\n",
    "- **MSELoss**：梯度与误差线性相关（\\(2(y_i - p_i)\\)），在分类任务中可能导致梯度饱和（如Sigmoid输出接近0/1时）。\n",
    "\n",
    "#### **(4) 异常值鲁棒性**\n",
    "- MSELoss对异常值敏感（平方放大误差），而交叉熵损失对概率偏差更敏感。\n",
    "\n",
    "---\n",
    "\n",
    "### **4. 选择建议**\n",
    "- **多分类任务**：优先使用`CrossEntropyLoss`（PyTorch中结合Softmax，无需手动激活）。\n",
    "- **二分类/多标签分类**：使用`BCELoss` + `Sigmoid`。\n",
    "- **回归任务**：使用`MSELoss`或`HuberLoss`（对异常值鲁棒）。\n",
    "- **避免**：用MSELoss处理分类问题（概率拟合不合理，梯度不稳定）。\n"
   ]
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "# 唐一旦老师的代码\n",
    "由于老师写的比较优美，我这边直接 copy 过来，然后加一点注释"
   ],
   "id": "dc1cc7993884eda1"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-18T07:56:36.177509Z",
     "start_time": "2025-08-18T07:56:25.571802Z"
    }
   },
   "cell_type": "code",
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "from torch.utils.data import DataLoader, random_split\n",
    "from torchvision import datasets\n",
    "import torchvision.transforms as transforms\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "torch.manual_seed(12046)\n",
    "\n",
    "dataset = datasets.MNIST(root='./datasets/mnist/', train=True, download=True, transform=transforms.ToTensor())"
   ],
   "id": "a861c20225199ad0",
   "outputs": [],
   "execution_count": 1
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-18T07:56:38.794819Z",
     "start_time": "2025-08-18T07:56:38.752549Z"
    }
   },
   "cell_type": "code",
   "source": [
    "train_set, val_set = random_split(dataset, [50000, 10000])\n",
    "test_set = datasets.MNIST(root='./datasets/mnist/', train=False, download=True, transform=transforms.ToTensor())\n",
    "\n",
    "train_loader = DataLoader(train_set, batch_size=500, shuffle=True)\n",
    "val_loader = DataLoader(val_set, batch_size=500, shuffle=True)\n",
    "test_loader = DataLoader(test_set, batch_size=500, shuffle=True)"
   ],
   "id": "37a4c13446400b3c",
   "outputs": [],
   "execution_count": 2
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-18T07:56:44.339655Z",
     "start_time": "2025-08-18T07:56:44.330538Z"
    }
   },
   "cell_type": "code",
   "source": [
    "model = nn.Sequential(\n",
    "    nn.Linear(784, 30), nn.Sigmoid(),\n",
    "    nn.Linear( 30, 20), nn.Sigmoid(),\n",
    "    nn.Linear( 20, 10)\n",
    ")"
   ],
   "id": "574727a7e4167e12",
   "outputs": [],
   "execution_count": 3
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T08:46:26.367322Z",
     "start_time": "2025-08-16T08:46:26.362928Z"
    }
   },
   "cell_type": "code",
   "source": [
    "eval_iters = 10\n",
    "\n",
    "def estimate_loss(model):\n",
    "    re = {}\n",
    "    # 将模型切换为评估模式\n",
    "    model.eval()\n",
    "    re['train'] = _loss(model, train_loader)\n",
    "    re['val'] = _loss(model, val_loader)\n",
    "    re['test'] = _loss(model, test_loader)\n",
    "    # 将模型切换为训练模式\n",
    "    model.train()\n",
    "    return re\n",
    "\n",
    "@torch.no_grad()\n",
    "def _loss(model, dataloader):\n",
    "    # 估算模型效果\n",
    "    loss = []\n",
    "    acc = []\n",
    "    data_iter = iter(dataloader)\n",
    "    for t in range(eval_iters):\n",
    "        inputs, labels = next(data_iter)\n",
    "        # inputs: (500, 1, 28, 28)\n",
    "        # labels: (500)\n",
    "        B, C, H, W = inputs.shape   # 第一个 B 是批次大小\n",
    "        logits = model(inputs.view(B, -1))\n",
    "        loss.append(F.cross_entropy(logits, labels))\n",
    "        # preds = torch.argmax(F.softmax(logits, dim=-1), dim=-1)\n",
    "        preds = torch.argmax(logits, dim=-1)\n",
    "        acc.append((preds == labels).sum() / B)\n",
    "    re = {\n",
    "        'loss': torch.tensor(loss).mean().item(),\n",
    "        'acc': torch.tensor(acc).mean().item()\n",
    "    }\n",
    "    return re"
   ],
   "id": "41f6d8f8947c1627",
   "outputs": [],
   "execution_count": 11
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T08:46:28.466051Z",
     "start_time": "2025-08-16T08:46:27.680421Z"
    }
   },
   "cell_type": "code",
   "source": "estimate_loss(model)",
   "id": "29814b78a60b09e1",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'train': {'loss': 2.3743958473205566, 'acc': 0.10120000690221786},\n",
       " 'val': {'loss': 2.3680973052978516, 'acc': 0.10480000078678131},\n",
       " 'test': {'loss': 2.3629441261291504, 'acc': 0.10699999332427979}}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 12
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T08:48:19.946487Z",
     "start_time": "2025-08-16T08:48:19.941953Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def train_model(model, optimizer, epochs=10, penalty=False):\n",
    "    lossi = []\n",
    "    for e in range(epochs):\n",
    "        for data in train_loader:\n",
    "            inputs, labels = data\n",
    "            B, C, H, W = inputs.shape\n",
    "            logits = model(inputs.view(B, -1))\n",
    "            loss = F.cross_entropy(logits, labels)\n",
    "            lossi.append(loss.item())\n",
    "            if penalty:\n",
    "                w = torch.cat([p.view(-1) for p in model.parameters()])\n",
    "                loss += 0.001 * w.abs().sum() + 0.002 * w.square().sum()\n",
    "            optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "        stats = estimate_loss(model)\n",
    "        train_loss = f'{stats[\"train\"][\"loss\"]:.3f}'\n",
    "        val_loss = f'{stats[\"val\"][\"loss\"]:.3f}'\n",
    "        test_loss = f'{stats[\"test\"][\"loss\"]:.3f}'\n",
    "        print(f'epoch {e} train {train_loss} val {val_loss} test {test_loss}')\n",
    "    return lossi"
   ],
   "id": "e97d4a96be78967",
   "outputs": [],
   "execution_count": 13
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T08:49:25.312164Z",
     "start_time": "2025-08-16T08:48:49.339284Z"
    }
   },
   "cell_type": "code",
   "source": [
    "loss = {}\n",
    "model = nn.Sequential(\n",
    "    nn.Linear(784, 30), nn.Sigmoid(),\n",
    "    nn.Linear( 30, 20), nn.Sigmoid(),\n",
    "    nn.Linear( 20, 10)\n",
    ")\n",
    "\n",
    "loss['mlp'] = train_model(model, optim.SGD(model.parameters(), lr=0.01))"
   ],
   "id": "b339654d6943da19",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 0 train 2.319 val 2.316 test 2.312\n",
      "epoch 1 train 2.304 val 2.303 test 2.303\n",
      "epoch 2 train 2.301 val 2.300 test 2.300\n",
      "epoch 3 train 2.300 val 2.300 test 2.300\n",
      "epoch 4 train 2.300 val 2.299 test 2.298\n",
      "epoch 5 train 2.299 val 2.298 test 2.297\n",
      "epoch 6 train 2.299 val 2.298 test 2.298\n",
      "epoch 7 train 2.297 val 2.297 test 2.296\n",
      "epoch 8 train 2.297 val 2.296 test 2.296\n",
      "epoch 9 train 2.295 val 2.296 test 2.296\n"
     ]
    }
   ],
   "execution_count": 14
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T08:50:56.688053Z",
     "start_time": "2025-08-16T08:50:17.853803Z"
    }
   },
   "cell_type": "code",
   "source": [
    "model = nn.Sequential(\n",
    "    nn.Linear(784, 30), nn.ReLU(),\n",
    "    nn.Linear( 30, 20), nn.ReLU(),\n",
    "    nn.Linear( 20, 10)\n",
    ")\n",
    "\n",
    "loss['mlp_relu'] = train_model(model, optim.SGD(model.parameters(), lr=0.01))"
   ],
   "id": "48ae19d806365962",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 0 train 2.290 val 2.290 test 2.290\n",
      "epoch 1 train 2.262 val 2.261 test 2.261\n",
      "epoch 2 train 2.212 val 2.210 test 2.211\n",
      "epoch 3 train 2.122 val 2.122 test 2.117\n",
      "epoch 4 train 1.953 val 1.950 test 1.945\n",
      "epoch 5 train 1.720 val 1.711 test 1.707\n",
      "epoch 6 train 1.454 val 1.453 test 1.447\n",
      "epoch 7 train 1.208 val 1.224 test 1.205\n",
      "epoch 8 train 1.022 val 1.018 test 1.001\n",
      "epoch 9 train 0.886 val 0.878 test 0.875\n"
     ]
    }
   ],
   "execution_count": 15
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T08:51:32.625997Z",
     "start_time": "2025-08-16T08:50:56.692303Z"
    }
   },
   "cell_type": "code",
   "source": [
    "model = nn.Sequential(\n",
    "    nn.Linear(784, 30, bias=False), nn.LayerNorm(30), nn.ReLU(),  # 归一化层截距项没有必要\n",
    "    nn.Linear( 30, 20, bias=False), nn.LayerNorm(20), nn.ReLU(),\n",
    "    nn.Linear( 20, 10)\n",
    ")\n",
    "\n",
    "loss['mlp_relu_layer'] = train_model(model, optim.SGD(model.parameters(), lr=0.01))"
   ],
   "id": "ccfe9d5dd3d7796a",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 0 train 1.554 val 1.552 test 1.543\n",
      "epoch 1 train 1.143 val 1.145 test 1.135\n",
      "epoch 2 train 0.829 val 0.843 test 0.835\n",
      "epoch 3 train 0.613 val 0.624 test 0.629\n",
      "epoch 4 train 0.518 val 0.501 test 0.503\n",
      "epoch 5 train 0.403 val 0.423 test 0.422\n",
      "epoch 6 train 0.355 val 0.365 test 0.362\n",
      "epoch 7 train 0.320 val 0.320 test 0.338\n",
      "epoch 8 train 0.286 val 0.318 test 0.290\n",
      "epoch 9 train 0.279 val 0.285 test 0.294\n"
     ]
    }
   ],
   "execution_count": 16
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T08:51:54.330953Z",
     "start_time": "2025-08-16T08:51:54.192504Z"
    }
   },
   "cell_type": "code",
   "source": [
    "for i in ['mlp', 'mlp_relu', 'mlp_relu_layer']:\n",
    "    plt.plot(torch.tensor(loss[i]).view(-1, 10).mean(dim=-1), label=i)\n",
    "plt.legend()\n",
    "plt.show()"
   ],
   "id": "9e1d4d058b9df2e4",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ],
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQAAaZ1JREFUeJzt3Xd4U+X/xvF3ZvdktAUKlCEgIiDIcgCCgiKKuFBURMEFKuJEv+IW1J+4NyooorjAAaiIgCxBtqwie5SySvdMcn5/nLZQAW2h7em4X9d1rjbJSfLJITR3nvMMm2EYBiIiIiIWsVtdgIiIiFRvCiMiIiJiKYURERERsZTCiIiIiFhKYUREREQspTAiIiIillIYEREREUspjIiIiIilnFYXUBw+n4+EhARCQkKw2WxWlyMiIiLFYBgGaWlp1KlTB7v9xO0flSKMJCQkEBsba3UZIiIichJ27dpFvXr1Tnh7pQgjISEhgPliQkNDLa5GREREiiM1NZXY2NjCz/ETqRRhpODUTGhoqMKIiIhIJfNfXSzUgVVEREQspTAiIiIillIYEREREUspjIiIiIilFEZERETEUgojIiIiYimFEREREbGUwoiIiIhYSmFERERELKUwIiIiIpZSGBERERFLKYyIiIiIpap1GJm8ZCdP/7CeXUmZVpciIiJSbVWKVXvLgsfr4605m9mTnMWERdu4pFUMQ89rROvY8MJ90nM8rE9IZeuBdA5n5pGclUtKZh7JmXk0jQrmlnPiiAhyW/ciREREqgCbYRiG1UX8l9TUVMLCwkhJSSE0NLRUHtMwDOb/fZAP5m9l/t8HC6/v0DCSWqF+rE9IZdvBjH99jBA/J0PPb8Qt58YR7Fdtc52IiMhxFffzu9qGkaOtT0hl/IKt/LA6gTxv0cMRE+bPaVEh1Az2IzzQRUSgiwC3k2+W72b93lQAIoPcDDkvjohAN0kZuRzOyCUpMxefzyA0wEWov4vQACdhAS7qRQQSVzOI6FB/7HZbqb8WERGRikJh5CQkpmTzzYrdOOw2WtYJ5fSYUGoE+x13X5/PYPpfexk3a9N/tqAcj7/LTsMaQcSE+ePndOBy2nE77LiddvyO2txOO2GBbjrGRdK0djA2mwKMiIhUDgoj5cTj9fHNit38sHov/i47EYFuIoPcRAS5sdsgLdtDalYeqdkeDmfmsvNQJjuTMvH4Sn7Ya4X40aVxDbo0roHTbmfXYfOxdidlkZJl9mNpVTeMVvXCOKNuGKH+rjJ4xSIiIsWjMFIcf34IyTvB5wFvHvjyzJ9+oRBaB0JjIKQOhERDYKR5fSm0THi8PnYfzmLbwQwOpOWQ6/WR6/Ed+enxkePx5v809122I4nsPF+JnifI7QCg4B/YYbNRM8SP2iF+RIf5ExXqD0BCchYJyVnsTcnmQFoOwf5Oagb7USvYj5ohfoQFOHHa7TjsNpx2W+HpJZ9hYBhm/xuH3U6NIDOI1Qh2UzPYj4ggN+EBLgLdjmK36BiGccJ9DcMgM9dLRo6HiCA3Lke1HgwmIlLhFffzu3r3ulz9Oez+s/j7250QEGFu7mBwuMHpBocfuPwhsAYER0FQLQiube7nDDBvcwWC0x9cAThdgTSsEUjDmkHFfuocj5cVO5JZtOUgS7Yl4bTbiI0IpH6NQOpFBBDi72TD3jTW7knhrz0p7D6cRUau95jHScvx/OdppeT8EUOb96cX/9j8C7fDTligi1B/Jy6HHbvNhtNhw26zkevxkZHrISPHQ3qOh+w8H35OO0F+TgJcDgLdDnyGQUpWHilZeYV9ehx2G3XDA2hQI5AGNQIJD3CTlm22QKVk5ZGWnVe4n8thL/xZ8JgBbvOnIz/4HJ3IbTYbdhvY83/6OR2E+DsJ9ncS7Ock0O0kK89LevaRunM8PnyGgcdr4PX58OYHNLfDfF6Xw55/Ks6G22nH7XDgctgwME/5eQ0Dr8/cPF4Dj8/A4/Ph8RqFtbuddlz5j2ejIBebNRbc7j7q9F7Bab+Cn9l5Pg6m5+RvuSRn5uKw2/BzOorcz2m35QdPO06H+XtBEHXkB1GzRh+5HvOnw2bDz2W+LrfTPN55/wjZtvxj6ee043/Uvq78Y+Jy2Mn1+jickcuh/L5XKVl5+DkdBPo5CHI7CfJzEOBy4F+42fFzOsjz+sjM9ZKV5yUr14vXZ5j7ue0Eus33kg3wFBxjnw9fQba3Ffy7m68rO89LjsdHdp6XPK8Pu+3IMXDYbRgGeH0GeV6f+W9mGPg5jzxPgNuB22Ev8r3FMCDX4zPry6/RZxiF78NAl5NAP8e/Buxcj4/U7Dx8PuOYf5+COnz5r8/ttBPs5yz107oer/kF6eh/W58PnPnvS7fDjstpw+2w4zzOazn6C4V53BxF3nc6DV19Ve+WkT/egeRd4HCawcLuArsDsg5D2l5I3QtpCZC2DzxZpfe8ANjMgBIQAZFx+Vtj86c72PzLaLObm8PPbJ0JiTHDTzEczsglLdtT5Lo8n4+DaTkkpmazLzWbxJQcAOqE+1MnPIA64QHUDvEjPcfDwbQcDqTncCAth9RsDz6fkf+H3IfHZ2DL/xC05X9o53p9JGXkkpSRy8H0XA6l53A4M/eYDsGlcuRs5h93karGabcR4HYUhhqH3UZ6tofU7LwSt4y6HDbCAtxEBLoIC3Dh8RlHWl29PvI8Bj7DwJffuukzDFwOO/6ugsBotqxm5HhIzfaQnlOyGpx2W+FjOR02MnO8ZOR6ONEZaofdRqi/k/BAN2EB5mABp8NOZq6H9BwvmTkeMvODpkF+qyxmlnQ5zDDtcpjP5fEahcEvO9d8ve6jAntBYDYDth2H3fw7VviFwGfg8Zqv1W6zYbOZ9dltNnz5Xxx8xpHWYfM2CoNrQbgveP22/C9eRx9/h+1ICDe/pEF2nhlYs/O3PK+R/28D3vznch8V3gt/5r9ul8MMdT7DfA0FX5AMzJbxgi+BBQH2yBcoM8yO7d+KplEhJXqf/RedpilteVlmSMlMgqwkyM0Eb665eXIgLxMyD0H6fsjYb/7MTjHvl5cFnmxzH5/nv5/r3wTWNE8fBUeZvwcVbLUhvD5ENIDQumaosljBt6DDmbkkZ+aRmp1X+J/d/A9t/sEM9nMS5Ge2Ovi7HOR4vGTm5m85Hmw2G+H5f1DDA134Ox3sT8thx6EMdhzKZPuhDNJzPIT6m/uEBjgJ9nNhtx35JpznNb/FZeWa30ozj/p2CuS3NNgwDPM/bsEfaZ/P/KacnuMhLdvcsvK8BLgcBPk5CMqv3c9px5X/TbXgVFbh83qMY1oJCn632cw/2ke3FhX8QTG/9drxGeaHSJ43f/MU/WNc8McxJ898zJz8b/a53iPfYH35fzBrBLmpEexHzWA3EYHuwsfOOerUoNcwvwEX1O/LbwkoaFEwDPL/gNpwHvXHr+Bxcj1mYC34gCho/TAMCp8nx+MjJ8+s75/9p4LcDiLyT/mF+rvI9frIzPWQmWP+O2Tlesn2eI8bdN1Os/XLYbeRnf9hVNK/cO78us2WHjs+g6P+sPuwH9VK5LTbsdshJ89X+L7y/kt/MIfdRqDLgZ/LgcOOeZ9cb4n6kBV8kFjNZjOPVcGHeK63+EGl4IP7ZPrOSdn59q4unFU/olQfU2GkovLmHQkoeZmQcQCStkHSVnM7vM0MLoYBhs/8mZcBaYlm8CkOuxPCYs3WFP+wo7Zw8/RRQStLcJS5FbO1RSovT/7phoo4nNznM8jzmSGm4Jt5cXh9R06puJ12/J3HnhowDIMcjxkUABwFYdFmK/zGW/AX0MAo7Bt1KgqCo/mYR+rwd534NEyuxwxc2Xnmz6yjvhmH+DvzpwdwEeznzD9VZBQGba/PyP+3Nb/9mkHMx+HM3CNfBLLycB4VDv2c9sIgVfCN3wZmmM0/RZXj8WEYBsF+5vOG5J+mDHA7Ck99/vNYe/IDbE7ekcfJ9njxeA0C3Q6C/czTnQEusx+ZN7+1puAUVmp2HoczcknOyiMlM49cr6/wy0qQn4NAtxNn/vMWnNExDPKD+pEvHe7891FA/qk8l8NeuI/5RcBbpEXA4zMK+74dCZrm+6MgjBe0gtiOagGx2470nys43erzQa7XS06ej2yP+dNrGGbIdTkKw67PyD9WHrMun8/Av6BV7Ki6C96nBc/l8R3Vt9DrIy8//Be8Pq/v2FOLQJEvgT6fgd1uw2GnyGvuGBdJeGDpfh4ojFQ1hmG2yqQlmKePMvZDxkHIPGj+TEs0O+Mm7zQ74paEX9hRLSy1IKye2cpSsIXFmqeTdD5XRERKQB1YqxqbDYJqmFt0qxPv5/Oa/V0O7zCDSnbKkS0zCdL3mcGl4KcvD3JSzC1py4kf9+h+KyHRUKMx1GpubjVPMzvpioiInASFkarG7jBbNsLq/fe+Ph9kJx/VwnLA7OuSsutIK0vyTvN6bw4k7zC3f7LZzRaUo0cSBdWG8Nj8TrmNzOvUsiIiIsdRrcPIy8teZvWB1Tze6XGaRjS1upzyZ7eb86cERgKnnXi/vOwjLSlpeyE1AQ5ugv0b4MAGs9Xl8HZzOxF3sNmaEtMa6rSFOmdBVEtwaGI2EZHqrlqHkTUH1rBy/0o2Hd5UPcNIcbn8zVE6EQ2Ovc0wzKCStDV/JFF+60r6PrMVJWmrOXw6Nx32rja3FZ+Y93X4mZPLuQLyt8AjoSWqJdQ+3TwNpFNAIiJVWrUOI43CG7Fi/wq2pmy1upTKy2bL70sSfeJ9PDlmH5YDGyFhJSSsMH9mp5ijh/718e1QsxnEdoD6nSC2o3naR6d8RESqjOodRsIaAbA1WWGkTDn9oNZp5nb6ZeZ1hmEGkfQD5oRyBcOdsw7DgXjYtw72r8u/nH86aMVE876BNcyAEtHwyFazCUSdYT6XiIhUKtU6jDQOawyglhEr2GxmC0dkoxPvYxhmP5WEFbDzD9i11GxRyTwEOxeZ29HsLnOkUb32ULcd1G5hdqD1Cy7b1yIiIqekWoeRRuHmB+HO1J3k+fJw2dWZskKx2czZZkP7QPM+5nWeHNi31pworqDT7OHtZktKVlL+KaAVRR8nOApqNDGHIDc4BxqeY/ZVERGRCqFah5GowCgCnYFkejLZlbqrMJxIBeb0M1s96rYrer1hmKFkz/L8bQUc+jt/iv595rZjISz/2Nw/spEZTKJbHZnYLTzWnKlWRETKVbUOIzabjUZhjVh7aC1bUrYojFRmNtuRBQdbXXXk+qzDcGirOaHb3tWwfQEkrjky/f4/BdaAhudB4wugSY/izdciIiKnpFqHETBP1aw9tJYtyVu4sMGFVpcjpS0gAuq1M7czrzGvy04x+6DsWAiHtuRP8rbLPM2TeQjWTzM3yD+10wWizzS3qNPBHWTVqxERqZKqfRhpHK5OrNWOfxic1svcjpaTbvY92ToHNs+GPcvMyd0ObjpqJ5sZUFr0hdbXmaN4RETklFT7MKLhvVLILxjqdzS3bo9AVjJs+90cwZO4BhL/MvueHIyH+fEw//+g3tnQegC07J8/k62IiJRUtQ8jBcN7t6dux+vz4rAXb/lyqQYCws15UQrmRgFI2wc7FsDqKbD5V9j9p7nNeMg8ndPsEmh+iTn3iYiIFIvNMAzD6iL+S3GXID4ZXp+XDp91INeXy4wrZhAbGluqjy9VWNo+WPs1rP7cbDU5Wu3ToVE3qN/ZDClBNS0pUUTESsX9/K72LSMOu4O4sDjiD8ezNWWrwogUX0gUdB5mbknbIH4mxM+AHYtg/3pz++Ntc9+ap0HDc+G0iyHufK23IyJylGofRsDsNxJ/OJ4tKVvoGtvV6nKkMoqMg853mVtmEmz5zQwlOxeboaSgI+yyj8wFARtfAKf1hsbdNXxYRKo9hRGOzMS6JXmLxZVIlRAYac51UjDfSWaSGUo2zzZbT9ISYOOP5gbmhGv1O5mndOLOh5paQVpEqheFEY6MqNmW8h8ryIqcjMBIczr75n2gz8vmyJz4n+DvnyFhlTnPyV+74K+vzP3jusI590DjHlqdWESqBYURis41YhgGNn0ASFmx2SCmtbl1e9ic22TPsiOTsG1fCNvmmVvUGdDlbnPYsNNtdeUiImVGYQSoH1Ifp81JRl4G+zL3ER0UbXVJUl34BZujbhp1My8f3gF/vAMrPjEXBJx6O/w4EuLyp6hvfIG56J8Cs4hUIXarC6gIXA5X4SgaTX4mlopoABePhZHroMdoCImBvAzY9BPMfAjebA+vtTY7wnrzrK5WRKRUKIzkK5j8TNPCS4UQEAHn3Q/3rYfb50PPp8y+JA43JO+AH++DtzrCuqnmisUiIpWYTtPkiwuLA2BLikbUSAVit0PMmeZ27gjIzYSVk2DeC+ZKxF/dDHXaQvfHzA6vdn2/EJHKR3+58hV2YtVpGqnI3IHQ8Ta4dxV0fQRcQebaOZ9dBW+0hQWvQMZBq6sUESkRhZF8BcN7t6RsoRLMkC/VnV8IdB9lhpJOw8AvDA5vh1+fhJebw9e3misQi4hUAgoj+RqGNcSGjZScFJKyk6wuR6R4gmtD7+fh/o1w+VtQtx348sw1c945B769zQwpIiIVmMJIvgBnAHWC6wDqxCqVkDsQ2t4AQ3+D2+bB6f0AA9ZMgTfaw4wHIX2/1VWKiByXwshR1G9EqoQ6beCaiXDbXHNeEl8eLH0f3mhndn7VaUgRqWAURo6i4b1SpdRpCzdOhZu+h5g2kJMK3w2DyddAaoLV1YmIFFIYOYqG90qV1KgrDJltzlXicMPfv8DbnWD1F2olEZEKQWHkKE0jzNVS1x9aj8fnsbgakVLkcJrzlNz+u9likp1iTjX/ThdY/DZkHLK6QhGpxhRGjtI8sjkh7hDSctNYe3Ct1eWIlL7aLeDWX+GC/4HTH/avh59Hwbjm8OUg2DpXrSUiUu4URo7itDvpUqcLAAsTFlpcjUgZcTjh/Afh/ni45P/MFYS9ubB+GnxyOXx6BSQqjItI+VEY+Ydz6pwDwILdCyyuRKSMBYRDh6HmqZvb50P7W80+JVvnwHvnwXfDIS3R6ipFpBpQGPmHc+ueC8C6Q+s0+ZlUHzFnwqXjYNhSc44SwwcrP4XXz4K5L0BOutUVikgVpjDyD7UCa9E8sjkGBosSFlldjkj5iowz5yi55Weo2x7yMmDu8/DGWbDsI/CqY7eIlD6FkeMoOFWzcI/6jUg1Vb8TDPkVrvoIIhpC+j748T54pzNsnK5OriJSqhRGjqPgVM2ihEX4DJ/F1YhYxGaDM66EYX9C7xcgIBIOboIvrocvb9T08iJSahRGjqN17dYEu4JJyk5iw6ENVpcjYi2nGzrdYa4QfO59YHfChh/grQ7w19dqJRGRU6Ywchwuu4tOMZ0AmL9nvsXViFQQ/mHQ80kYOgeiWkHWYfjmVphyA6Tts7o6EanEShRGxowZw9lnn01ISAi1a9emX79+xMfH/+f9vvrqK5o3b46/vz+tWrVixowZJ11weTmnrvqNiBxXzJlw2xzo9qjZSrLxR3i7I6z91urKRKSSKlEYmTdvHsOGDeOPP/5g1qxZ5OXlcdFFF5GRkXHC+yxatIjrrruOW2+9lZUrV9KvXz/69evH2rUVe1Klgn4jaw6uISUnxeJqRCoYhwu6PWyuDByd30ry9WD4ajBkaki8iJSMzTBO/oTvgQMHqF27NvPmzeP8888/7j7XXnstGRkZ/Pjjj4XXderUiTZt2vDuu+8W63lSU1MJCwsjJSWF0NDQky23xK747go2J2/mpa4v0bth73J7XpFKxZMLv78E818GwwvBUdD3NWh2sdWViYjFivv5fUp9RlJSzBaDyMjIE+6zePFievbsWeS6Xr16sXjx4hPeJycnh9TU1CKbFQpaRzQbq8i/cLrhgsdgyCyoeZo5DPjzATDxMtihuXpE5L+ddBjx+XyMGDGCc845hzPOOOOE+yUmJhIVFVXkuqioKBITTzzN9JgxYwgLCyvcYmNjT7bMU1LYbyRhIafQgCRSPdRtZ04t33m42Zdk2zz4+GKYcClsV6AXkRM76TAybNgw1q5dyxdffFGa9QAwatQoUlJSCrddu3aV+nMUx1m1zyLAGcDBrIPEH/7vjroi1Z4rAHo9B3evgHaDwe6C7fNhQh+YfC1kJVtdoYhUQCcVRoYPH86PP/7InDlzqFev3r/uGx0dzb59RYf97du3j+jo6BPex8/Pj9DQ0CKbFdwONx1jOgIwe+dsS2oQqZQiGkDfV+GelUcW4Nv0E4zvAQc3W12diFQwJQojhmEwfPhwpk6dym+//UZcXNx/3qdz587Mnl30g3zWrFl07ty5ZJVapKDj6vSt03WqRqSkwmPNBfiGzIbQenBoM4y/ADYr3IvIESUKI8OGDWPSpElMnjyZkJAQEhMTSUxMJCsrq3Cfm266iVGjRhVevvfee/npp594+eWX2bhxI08++STLli1j+PDhpfcqytAF9S8g0BnIrrRdrD6w2upyRCqngrlJ6nWA7BT47Cr44x3N3ioiQAnDyDvvvENKSgrdunUjJiamcJsyZUrhPjt37mTv3r2Fl7t06cLkyZN5//33ad26NV9//TXTpk37106vFUmAM4CeDczRQD9s+cHiakQqseDacPOP0GYgGD746RFzbhLNSyJS7Z3SPCPlxap5RgosTljMbbNuI9Qdypxr5uB2uMu9BpEqwzBg8Zsw6wlzXpKQOnDFO9Com9WViUgpK5d5RqqLDtEdqB1Qm9TcVObv1lo1IqfEZoMud8OtsyCyMaQlwCeXw0+PQl621dWJiAUURorBYXfQp1EfAH7YqlM1IqWiXju4Yz60v8W8/Mdb8GFPSN377/cTkSpHYaSYLm18KQDzds/TWjUipcUdBJe+AtdNgcCakPgXfHSRhv+KVDMKI8V0WsRpNItohsfn4eftP1tdjkjV0qw3DJ0NkY0geSd81AsSVlpdlYiUE4WREujbuC8A32/53uJKRKqgiIZwyy8Q0wYyD5rTyG+ZY3VVIlIOFEZK4JK4S7Db7Kw+sJqdqTutLkek6gmuZQ7/jesKuenw2dWwesp/309EKjWFkRKoFViLTjGdAPhx648WVyNSRfmFwMCvoOUV4MuDqbeZI228HqsrE5EyojBSQpc2Mjuy/rDlB7w+r8XViFRRTj+48iM4/yHz8h9vwaf9IOOgpWWJSNlQGCmhHvV7EOIOYXf6bn7b9ZvV5YhUXXY7XPAYXPMpuIPN1X/f7wYJq6yuTERKmcJICQW6Arm++fUAfLDmAy2eJ1LWTr/MXGgvshGk7ILxPc3ZW3PSrK5MREqJwshJGNhiIAHOADYkbWBRwiKryxGp+mo3h6FzoFkfsx/Jwlfhjfaw6nPw+ayuTkROkcLISYjwj+Cq064C4IO/PrC4GpFqIiAcBnwG130BEXGQngjT7oAPL4R966yuTkROgcLISRp0+iCcdifL9y1n5X5NziRSLmw2aHYxDFsCPZ80+5LsWQYf9oJtWjdKpLJSGDlJUUFRXN74cgDG/zXe4mpEqhmnH5x7HwxfBg3Ogdw0mNQf1mtCQpHKSGHkFAw+YzB2m53fd/9OfFK81eWIVD+hMXDDt9D8UvDmwleDYPkEq6sSkRJSGDkFDUIb0KtBLwA+/OtDi6sRqaZc/nD1RDhrEBg++OFemPcSaKSbSKWhMHKKbm11KwA/7/iZHak7LK5GpJpyOKHva3D+g+blOc/C5wMg/YC1dYlIsSiMnKJmkc3oWq8rPsPHxHUTrS5HpPqy2eCC/0GfceBww6af4J3OsOkXqysTkf+gMFIKBrUcBJhTxKfkpFhcjUg1d/at5pwktU+HjAMw+WqYfj/kZlpdmYicgMJIKWgf1Z5mEc3I9mbzzd/fWF2OiESfYQaSjneal/8cDxP7Qk66tXWJyHEpjJQCm83GDaffAMDnGz/H49PqoiKWc/nDxWPN0TYBEeZ8JF/eCJ5cqysTkX9QGCklF8ddTKR/JIkZiczeOdvqckSkQJMeMPBrcAXClt/MWVs1hbxIhaIwUkr8HH5cfdrVAExaP8niakSkiHrt4dpPwe6Ctd/ATw9r6K9IBaIwUoqubXYtTruTVQdWsfbgWqvLEZGjNekJV7wL2GDp+zDvRasrEpF8CiOlqFZgLXo37A3ApA1qHRGpcFpdBRfnh5C5z8OS962tR0QAhZFSV9CR9eftP7M/c7/F1YjIMTreBl0fMX+f+SCs1BcHEaspjJSyljVa0rZ2Wzw+D1Pip1hdjogcT7dHoNMw8/fv74a131pbj0g1pzBSBga2GAjAV/FfkeXJsrgaETmGzQa9noN2N5vr2Xw7FOJ/sroqkWpLYaQM9Kjfg7rBdTmcc5gv47+0uhwROR6bzZw6vtXV4PPAlzeZgUSjbETKncJIGXDandx+5u2AuZpvZp6moRapkOwO6PcONL8UvDnw+bXwfldY+RnkZVtdnUi1oTBSRvo27ktsSCyHcw4zeeNkq8sRkRNxuOCqj6D9reDwg72r4bu7YFwL+PUpyM2wukKRKk9hpIw47U7ubG2ui/Hx2o9Jz9WaGCIVltMPLh0HIzdAzychLBaykmDBOPj2Np26ESljCiNl6JK4S4gLiyM1N5VPN3xqdTki8l+CasC598E9q8zWEocbNv4Ii163ujKRKk1hpAw57A7uan0XAJ+u+5SUnBSLKxKRYnE44YwrofdY8/KvT8H2hdbWJFKFKYyUsYsaXkST8Cak5aXxyfpPrC5HREqi/S1w5rVgeOHrwZCWaHVFIlWSwkgZs9vsDGtjTq40af0kDmcftrgiESk2mw0ufQVqtYD0ffD1LeD1WF2VSJWjMFIOetTvQYvIFmR6Mvl43cdWlyMiJeEOMlf8dYfAjoXw8yjw5FpdlUiVojBSDmw2W2HryBcbv+Bg1kGLKxKREqnZFC5/0/x96fvwZntY9Tn4vNbWJVJFKIyUk/PrnU+rmq3I8mTx8Vq1johUOi37weVvQ3AUJO+AaXfAO11g/fca+ityihRGysnRrSNT4qdwIPOAxRWJSIm1HWgO++35FPiHw4GN8OWNMO0utZKInAKFkXLUpU4X2tRqQ443h/F/jbe6HBE5Ge5AOHcEjFgD5z8INgesngzf3wM+n9XViVRKCiPlyGazMbztcAC+2vQViRkaJihSafmHwQX/gyvHg80OqybBj/cqkIicBIWRctYhugPto9qT58tT64hIVXBGf+j/gRlIVnwC00eqD4lICSmMlLOj+4588/c3JKQnWFyRiJyyVldBv3cBGyz/GGY8oBYSkRJQGLFA++j2dIrphMfn4f0171tdjoiUhtbXQr+3ARv8OR6+uRXysq2uSqRSUBixSEHryLTN09hwaIPF1YhIqWhzPfR/H+wuWPctTOoPmUlWVyVS4SmMWKRN7TZc2OBCvIaXB+Y9QHpuutUliUhpOPMauOEb8As1Z2z9qBcc3mF1VSIVmsKIhZ7o/AR1guqwM20nTyx6AkOd3kSqhkZd4ZafILQuHNwEH14ICausrkqkwlIYsVCYXxgvdX0Jp93JLzt+4Yv4L6wuSURKS1RLGPIrRJ1hLrI34VLY9rvVVYlUSAojFjuz1pnc3+5+AF768yXWHVxncUUiUmpC68DgmdDwPMhNg0lXmtPHi0gRCiMVwMAWA+lZvyd5vjzun3c/qbmpVpckIqXFPxQGfg0t+oI3F74aBMsnWF2VSIWiMFIB2Gw2njrnKeoG12VP+h6eXPSk+o+IVCUuf7h6Ipx1Exg++OFemP+yJkcTyacwUkGEukN5udvLOO1OZu2YxXdbvrO6JBEpTXYH9H0dzh1pXp79NMwda21NIhWEwkgF0rJGy8L5R8YsGcOutF0WVyQipcpmg55PwIVPm5fnjYV5L1pbk0gFoDBSwQxuOZizap9FpieTxxY8hsfnsbokESlt59wLPZ8yf5/znHnKRqQaUxipYBx2B8+f9zxBriBW7l/JR2s/srokESkL546AHqPN32c/DQtfs7QcESspjFRAdYPr8ljHxwB4Z9U7rD241uKKRKRMnHc/dDf/rzNrNMx7SQvsSbWkMFJBXdroUno17IXH8DBq/iiyPFlWlyQiZaHrQ9D1YfP3Oc/CZ1dB+n5raxIpZwojFZTNZuPxTo9TO7A221O38+3f31pdkoiUlW6j4NJXwekPW2bDO+fAljlWVyVSbhRGKrAwvzCGthoKwOcbP8dnqPlWpEqy2aD9YBg6B2q1gIz98OkV8OuT4PNaXZ1ImVMYqeAua3wZIa4QdqTuYMGeBVaXIyJlKep0GPobtBsMGLDgFZg/zuqqRMqcwkgFF+gK5IqmVwAwecNki6sRkTLnDoS+r0Kf/BAy7wVIVCd2qdoURiqBAc0HYMPGwoSFbE3ZanU5IlIe2t8CzfqALw+m3QHePKsrEikzCiOVQGxILN1iuwFqHRGpNmw2uPQVCIiAxL80MZpUaQojlcTAFgMB+H7L91rVV6S6CImCS/7P/P33l2DvGmvrESkjCiOVRIfoDjQJb0KWJ4tpf0+zuhwRKS9nXAkt+oLPA9PuBE+u1RWJlDqFkUrCZrMVto58vvFzvBruJ1I92GzQ5xUIrAH71sLcMVZXJFLqFEYqkT6N+hDmF8bu9N38vvt3q8sRkfISXAv65PcZWTAO5jwPhmFtTSKlSGGkEglwBnBl0ysBmLh+Iob+GIlUHy2vOLKOzbwX4Mf7NCGaVBkKI5XMdc2vw213s3zfcubummt1OSJSnro+lD//iA2Wfwxf3Qx52VZXJXLKShxGfv/9d/r27UudOnWw2WxMmzbtX/efO3cuNpvtmC0xMfFka67WooOiuanlTQD837L/I9erzmwi1crZt8LVE8Dhhg3fmwvr5aRZXZXIKSlxGMnIyKB169a89dZbJbpffHw8e/fuLdxq165d0qeWfENaDaFmQE12pu3k842fW12OiJS3lv1g4NfgDoHt8+HLmzQpmlRqJQ4jF198Mc8++yxXXHFFie5Xu3ZtoqOjCze7XWeITlaQK4h72t4DwLur3+VQ1iGLKxKRcteoKwz6DlyBsOU3mD5SnVql0iq3RNCmTRtiYmK48MILWbhw4b/um5OTQ2pqapFNirq8yeW0iGxBel46b6560+pyRMQKddvBVR+BzQ4rPjEX1hOphMo8jMTExPDuu+/yzTff8M033xAbG0u3bt1YsWLFCe8zZswYwsLCCrfY2NiyLrPSsdvsPNzhYQC+/ftb4pPiLa5IRCzR7GLo/YL5++yn4K+vra1H5CTYjFMYH2qz2Zg6dSr9+vUr0f26du1K/fr1+fTTT497e05ODjk5OYWXU1NTiY2NJSUlhdDQ0JMtt0p6YN4D/Lz9ZzpEd2D8ReOx2WxWlyQiVvhpFPzxNjj84Map0PAcqysSITU1lbCwsP/8/Lak40aHDh3YvHnzCW/38/MjNDS0yCbHN7LdSNx2N0sTlzJt8zSryxERq1z0LDS/FLw5MOESeKsT/PwYbP4V8rKsrk7kX1kSRlatWkVMTIwVT13l1Amuw+2tbwfg6cVPsyhhkcUViYgl7A7o/wE0uwSwwYENsPhNmHQlvBCn0zdSoTlLeof09PQirRrbtm1j1apVREZGUr9+fUaNGsWePXv45JNPAHj11VeJi4ujZcuWZGdnM378eH777Td++eWX0nsV1dyQVkPYkryFGdtmMHLuSCb2nkizyGZWlyUi5c0dCNd9DplJsHWuOcpmyxxI3Q3fDYeoM6B2c6urFDlGiVtGli1bRtu2bWnbti0AI0eOpG3btowePRqAvXv3snPnzsL9c3Nzuf/++2nVqhVdu3Zl9erV/Prrr/To0aOUXoLYbXaeOecZzo4+m4y8DO769S72pu+1uiwRsUpgJJzRHy5/E0b8BY0vAE8WfH2LZmyVCumUOrCWl+J2gKnuUnNTGTRzEJuTN9MkvAkTL55IqFvHS6TaS9sH754DGQfg7KHQ5/+srkiqiQrdgVXKRqg7lHd6vkPtgNpsTt7MyLkj8Rk+q8sSEauFRMEV75q///kBbJxubT0i/6AwUsVEB0Xzds+3CXAGsGTvEr7epE5rIgI06Qld7jZ//24YpOyxth6RoyiMVEHNIptx71n3AjBu+Tj2ZeyzuCIRqRAuGA0xbSDrMHw1CPats7oiEUBhpMoa0GwAZ9Y6k4y8DJ5d8iyVoGuQiJQ1p9ucPt4dDLv/hHe6wMTLIH4m+HRKV6yjMFJFOewOnur8FE67k7m75jJrxyyrSxKRiqBGY7jlJzj9cnNNm23z4PMB8GY7M5SIWEBhpAprEtGEIa2GAPD8kudJyUmxuCIRqRCiW8E1n8C9q6HLPeAfBklb4fPrYOHrWv1Xyp3CSBU3tNVQ4sLiOJR9iHHLx1ldjohUJOH14aJnYOQGOHsIYMCsx+GHe8CbZ3V1Uo0ojFRxboebp7o8BZir+2q6eBE5hjsILvk/c/Vfmx1WfGJOI5912OrKpJpQGKkG2tZuy7XNrgXgkd8fISE9weKKRKTCsdmg0x0w4HNwBZl9ST68CDIOWV2ZVAMKI9XEA+0foEVkCw7nHObeOfeS5dEqniJyHM16w60/Q2hdOLjJPGWjPiRSxhRGqgl/pz+vdX+NSP9INiZt5IlFT2i4r4gcX3Qrc8E9uws2/ggrP7W6IqniFEaqkZjgGF7u+jJOm5OZ22Yycd1Eq0sSkYoqpjVc8D/z95mPwKEt1tYjVZrCSDXTPro9D3d4GIBXVrzCwj0LLa5IRCqsLndDw/MgLwO+GaIRNlJmFEaqoWubXUv/pv3xGT4e+v0hDmYdtLokEamI7A5zgT3/MEhYAfNesLoiqaIURqohm83GYx0fo0VkC1JzUxm3TPOPiMgJhNWDS181f5//MmyZY2k5UjUpjFRTboeb0Z1HY8PGD1t/4M/EP60uSUQqqjP6Q+vrwPDBp/3g40tg3VSdtpFSozBSjZ1R8wyuaXYNAM/+8Sx5+sMiIidyyUtwxpVgc8COhfDVzfBqK5j3IuRpqgA5NQoj1dzdbe8m0j+SrSlbmbheo2tE5AT8QswVf0f8Bec/CEG1IG0vzHkOPuoFh3dYXaFUYgoj1VyYXxgPtH8AgPdWv6fZWUXk34XVNYf83rcerngfAmvA3tXwflfYPNvq6qSSUhgRLm10Ke2j2pPtzWbM0jFWlyMilYHTDa2vhdvmQZ225jo2k66E3/8PfD6rq5NKRmFEsNls/K/T/3DanMzdNZc5O9VbXkSKKTwWBv8EZ90EGPDbMzDxUtjwI3g9VlcnlYTCiADQOLwxN7W8CYCxS8dq7RoRKT6XP1z2BvR9HRxus4PrlIHwWmv4/SVI3291hVLBKYxIodvPvJ2YoBgSMhL4YM0HVpcjIpVNu0Fw93I4ZwQERELqbvjtWXjlDFj/ndXVSQWmMCKFAl2BhVPFf7zuY7ambLW4IhGpdMLrw4VPwcgNcMV7Zn8Sbw58e7vZ0VXkOBRGpIgLYi/g/Hrn4/F5eH7J81rZV0ROjssfWg+AW3+Fxj3AkwWfX69TNnJcCiNShM1m45EOj+Dn8GPJ3iX8tP0nq0sSkcrM4YSrPoTIxuZpmyk3gifX6qqkglEYkWPEhsQytNVQAF7880XSctMsrkhEKrWACLjuC/ALhV1/wIz7Qa2uchSFETmuwWcMpkFoAw5mHeTtVW9bXY6IVHa1TjNncMUGKz6BP96xuiKpQBRG5LjcDjePdngUgMkbJ7P6gDqeicgpanqh2bkV4OdR5vo26kMiKIzIv+hStwt9GvXBZ/h4+PeHSc9Nt7okEansutwD5z9kLri3biq8eTasmqzTNtWcwoj8q8c6Pkbd4LrsSd/Dc0ues7ocEansbDa44DEY+htEnwnZyTDtTpjUH1L3Wl2dWERhRP5ViDuEseeNxWFz8OPWH/lhyw9WlyQiVUGdNmYg6fkkOP1hy28wsS9kHLS6MrGAwoj8pza123BH6zsAeG7Jc+xK22VxRSJSJThccO59cMcCCK0Hh/6GT6+A7BSrK5NypjAixTK01VDOqn0WGXkZPPL7I+T58qwuSUSqippN4abvILAmJK6ByddCbqbVVUk5UhiRYnHYHYw9bywhrhDWHFzDu6vftbokEalKajaBG6eCXxjsXAxfanK06kRhRIotJjiG0V1GAzD+r/Es37fc4opEpEqJORMGfgmuQNj8K3x5ExzaYnVVUg4URqREejfszeWNL8dn+Bg1fxSpualWlyQiVUn9TnDtJLC7YNNMeKMdTB4AW+dp+G8VpjAiJTaq4yhiQ2LZm7GXZxY/o8X0RKR0NekBN0+HJhcChhlKPrkM3ukC2xdaXZ2UAYURKbEgVxAvnPcCTpuTn7b/xA9bNdxXREpZ/Y5ww9cwfBmcPcQ8dbN/PXx2NSSstLo6KWUKI3JSWtVqxV1t7gLguT+eY1eqhvuKSBmo2RT6vAwjN0BcV8jLgM+ugcPbra5MSpHCiJy0W864hXZR7cj0ZPLI/EfI9arnu4iUkYBwsy9JVCvI2A+TroSMQ1ZXJaVEYUROmsPuYMy5Ywhxm8N9h/4ylKTsJKvLEpGqyj8UBn4FYbFwaDN8rvlIqgqFETklMcExvNLtFUJcIazYv4Lrp1/P34f/trosEamqQmPghm/APxx2/wnf3AqeHKurklOkMCKnrGNMRyb1mURsSCx70vdww4wb+H3371aXJSJVVa1mcN0X4PCD+Bnw4UWQtM3qquQUKIxIqWgU1ojJl0zm7OizyfRkMnz2cL6M/9LqskSkqmrQGa7/AgIiYe8qeK8rbPjR6qrkJCmMSKkJ9w/nvZ7vcWXTKzEwGLt0LDtSd1hdlohUVY0vgDvmQ70OkJMCUwbCT49qGvlKSGFESpXL4eKJzk9wTt1zyPPlMWbpGE2KJiJlJ6weDJ4BnYebl/94Cyb0gdS91tYlJaIwIqXOZrMxqsMoXHYXC/csZM6uOVaXJCJVmcMFvZ6DAZPNhfZ2L4X3u8LOP6yuTIpJYUTKRIPQBgxqOQiAF/98kWxPtsUViUiV17wP3DYHarWA9H0w4VJY+oHWtKkEFEakzAxtNZSowCj2pO/ho7UfWV2OiFQHNRrDkF/h9H7gy4MZD8C0O2HfOoWSCkxhRMpMoCuQB89+EIAP//qQXWmaMl5EyoFfMFw9AS58Gmx2WP25ucjeq63gx5Gw6RfNTVLBKIxImbqowUV0jOlIri+XF/980epyRKS6sNngnHvhpu+h6UXg9IeUXbDsQ5h8NbzVAXYvs7pKyacwImWqoDOr0+Zk7q65/LBFK/yKSDmKO8+cQv6hbXDdFGh/CwTVMhfa+6gXzB8HPp/VVVZ7CiNS5hqHN2bwGYMBeHzh48zdNdfSekSkGnIHQrPecOkrMHwZtLwCfB6Y/RR82k9DgS2mMCLlYnjb4Vza6FK8hpf7597Pkr1LrC5JRKqrgHC46mO47E1wBcK2eWafkj8/BG+e1dVVSwojUi7sNjvPnPMMF8ReQK4vl7t/u5vVB1ZbXZaIVFc2G5x1I9z+O0S3gqwkmD4S3mwPq78An9fqCqsVhREpN067k5e6vkSnmE5kebK489c7iU+Kt7osEanOajaFIb/BxS9CUG2zL8nU282Wks2/Wl1dtaEwIuXK7XDzWvfXaF2rNWm5adw1+y5Sc1OtLktEqjOnGzreDveugh5PgH84HNgIn10DO3VKuTwojEi5C3QF8nbPt2kQ2oD9mft5ednLVpckIgLuIDhvJNy7GppfCoYXvrkVMpOsrqzKUxgRS4S6Q3mqy1MAfPv3tyxKWGRxRSIi+QLCod87ENnInJvku2GavbWMKYyIZdpFtWNAswEAPLXoKTLzMi2uSEQkn3+oOYurww3xM2DJu1ZXVKUpjIilRrQbQUxQDAkZCby24jWryxEROSKmNfR63vz9l8dhz3Jr66nCFEbEUkGuIJ7s/CQAn2/8nJX7V1pbkIjI0c4eAi36movufTVY/UfKiMKIWK5L3S70a9IPA4PRC0eT49UCViJSQdhs5uRo4fUheQe81hpmjdaMraVMYUQqhAfaP0DNgJpsT93OqPmjyNMsiCJSUQSEw7WfQc3TICcVFr5mrgA8bRgc0FxJpUFhRCqEML8wnjvnOVx2F7N2zOKeOfeQ7cm2uiwREVPMmXDXErjuC6jf2Txts2oSvN0ZfhoF2Zov6VQojEiF0aVuF9644A38Hf4s2LOAO3+9k/TcdKvLEhEx2e3Q7GK45Se4dRacdrE5F8kfb8ObZ8NfX2sI8ElSGJEK5Zy65/Dehe8R7Apm2b5lDP1lKMnZyVaXJSJSVGwHuP4LuOEbiGwM6YnmBGkT+8KBTVZXV+kojEiFc1bUWYzvNZ5wv3DWHlrLkF+G6JSNiFRMTXrCXYuh+//A6Q/b58O758DcF8CjzvjFpTAiFVLLGi2Z0HsCkf6RxB+O5+3Vb1tdkojI8Tn9oOuDMGwJNL0IvLkw93l49zzYsdjq6iqFEoeR33//nb59+1KnTh1sNhvTpk37z/vMnTuXs846Cz8/P5o0acKECRNOolSpbhqHNy6cg2TiuomsPbjW2oJERP5NREO4/ku46iMIqgUH4+Hj3jD1TtjwA2QctLrCCqvEYSQjI4PWrVvz1ltvFWv/bdu20adPH7p3786qVasYMWIEQ4YM4eeffy5xsVL9dK/fnYvjLsZn+Bi9aLSG/IpIxWazwRlXwrClcNZN5nWrJ8OUG+ClxvBmB/jhXkhYZWmZFY3NME6+66/NZmPq1Kn069fvhPs8/PDDTJ8+nbVrj3yrHTBgAMnJyfz000/Fep7U1FTCwsJISUkhNDT0ZMuVSupw9mH6fdePpOwk7mp9F3e2udPqkkREimfHYvjrS/PngQ1Hrne44bI3oPUA62orB8X9/C7zPiOLFy+mZ8+eRa7r1asXixef+DxaTk4OqampRTapviL8IxjVYRQA7695n02H1VNdRCqJBp3h0ldg2B/w0DYY8Dk07WX2K5l6O8x6Anxeq6u0XJmHkcTERKKioopcFxUVRWpqKllZWce9z5gxYwgLCyvcYmNjy7pMqeB6NexF99jueAwPoxeOxuPzWF2SiEjJBEZC80vMidPOe8C8buGr8MVAyEmztDSrVcjRNKNGjSIlJaVw27Vrl9UlicVsNhv/6/Q/QtwhrDu0jvfWvGd1SSIiJ8duhx6PQ//x4PCDTTNh/IWwb53VlVmmzMNIdHQ0+/btK3Ldvn37CA0NJSAg4Lj38fPzIzQ0tMgmUjuwNo90eASAd1e/y7d/f2txRSIip+DMq2HwTAiOMvuTvHc+zHm+Ws5PUuZhpHPnzsyePbvIdbNmzaJz585l/dRSBV3W+DJuPeNWAJ5a/BRzd821tB4RkVNSrx3c/js06wM+D8x7Ad7rCruXWV1ZuSpxGElPT2fVqlWsWrUKMIfurlq1ip07dwLmKZabbrqpcP877riDrVu38tBDD7Fx40befvttvvzyS+67777SeQVS7dx71r1c3vhyfIaPB+Y9wKr9q6wuSUTk5IVEw4DP4KqPIbCm2Ury4YXwzRBYNxWyU6yusMyVeGjv3Llz6d69+zHXDxo0iAkTJnDzzTezfft25s6dW+Q+9913H+vXr6devXo8/vjj3HzzzcV+Tg3tlX/K8+Vx72/3Mn/PfELdoXx68ac0Cm9kdVkiIqcmM8lcBXjNF0euszuhQRdzYb4210NAuGXllVRxP79PaZ6R8qIwIseTmZfJ0F+GsubgGqICo3jvwvdoHN7Y6rJERE7drj9hw3ew6Wc4eNR0Bv5h0OVu6HgH+IVYV18xKYxItZCcncygnwaxNWUrwa5gXu72Ml3qdLG6LBGR0nNoC/z9CyyfeGTitIBIOHcEnD0U3IGWlvdvFEak2jicfZgRc0awYv8KHDYHj3Z8lGuaXWN1WSIipcvnNfuQzB0Dhzab17kCocE50Lg7NOoOtVuYU9JXEAojUq3kenN5avFTfL/lewBuaHEDD7R/AIfdYXFlIiKlzOuBNVPg9xfh8PaitwVHQ6OuEHe+uYXXt6TEAgojUu0YhsEHf33AGyvfAKB3w96MPW+sAomIVE2GAfvXw5bfYMsc2LEIPP+Y2Twizuz0ev6DlrSYKIxItfXT9p8YNX8UHp+HK5teyROdn8BWgZotRUTKRF427FoC236HbfNgzwow8te9ufJDaHVVuZekMCLV2i/bf+HB3x/EZ/i45YxbuK+d5rURkWomO9U8lbPoDQipA8P/BL/gci2hwqzaK2KFixpexBOdnwDgo7Uf8eFfH1pckYhIOfMPhe7/g/AGkJYAC8ZZXdEJKYxIldW/aX/ub3c/AK+ueJWvNn1lcUUiIuXM5Q+9x5i/L3oDkrZaW88JKIxIlXbzGTczpNUQAJ5Z/Azvrn4Xn+GzuCoRkXLU7BJofAF4c+Hnx6yu5rgURqTKu6ftPdzQ4gYMDN5a9RbDZg8jOTvZ6rJERMqHzQa9x5rTysfPgM2/Wl3RMRRGpMqz2Ww83OFhnu7yNH4OPxbsWcA1P17DXwf+sro0EZHyUasZdLjd/H3mI5CbYZ6y2TrXnNl19jOQddiy8jSaRqqV+KR4Rs4dyc60nTjtTp7s/CSXN7nc6rJERMpedgq80Q4yDhz/9iGzoV77Un1KjaYROY5mkc344tIv6Fm/Jx6fhycXP0l8UrzVZYmIlD3/MLjo2SOXnf5Q8zRociG0vxX8rPuyr5YRqZYMw+DeOfcyZ9ccmkU04/M+n+NyuKwuS0Sk7CVtM9e0Ca5d5rOyqmVE5F/YbDZGdx5NhF8E8YfjeWf1O1aXJCJSPiLjICSqQi2opzAi1VbNgJr8r9P/APhw7YesObDG4opERKonhRGp1i5qeBGXxF2Cz/Dx2ILHyPZkW12SiEi1ozAi1d6jHR+ldkBttqdu57UVr1ldjohItaMwItVemF8YT3Z5EoBJGybx0O8PsWDPAjw+j7WFiYhUExpNI5LvhaUvMGnDpMLLtQJq0adRH65tdi31QupZWJmISOVU3M9vhRGRfIZhsP7Qer7f8j0zts0gOScZgFB3KF/1/Yo6wXWsLVBEpJJRGBE5BXnePBbsWcAbq97g78N/c2atM5nQewIuu+YiEREpLs0zInIKXA4X3et35/XurxPiCmHNgTW8vuJ1q8sSEamSFEZE/kW9kHo8c84zAExYN4F5u+ZZXJGISNWjMCLyH3o06MHAFgMBeGzhYyRmJFpckYhI1aIwIlIMI9uNpGWNlqTkpPDgvAdZd3Admw9vZlfaLg5kHtAwYBGRU6AOrCLFtCttF9f8cA3peenH3FY/pD5f9v2SIFeQBZWJiFRM6sAqUspiQ2IZ120czSObEx0UTYRfBAHOAGzY2Jm2k/F/jbe6RBGRSkktIyKnaM7OOdwz5x7cdjff9ftOE6SJiORTy4hIOekW241OMZ3I9eUybvk4q8sREal0FEZETpHNZuOhsx/CbrMza8cs/kz80+qSREQqFYURkVLQNKIpV592NWCuceP1eS2uSESk8lAYESklw9oMI8QdQvzheL7d/K3V5YiIVBoKIyKlJMI/grta3wXAmyvfJC03zeKKREQqB4URkVJ0bfNriQuLIyk7iT7f9mHU/FHM3DaTlJwUq0sTEamwNLRXpJSt3L+Se3+7l8M5hwuvs9vsdI7pzBOdnyAmOMbC6kREyk9xP78VRkTKQJ4vj1X7VzF/z3zm757P5uTNAET4RfBi1xfpFNPJ4gpFRMqewohIBbItZRsP//4wG5I2YLfZufesexnccjA2m83q0kREyowmPROpQOLC4vjk4k+4vPHl+Awfryx/hfvn3U9GXobVpYmIWE5hRKSc+Dv9eeacZ3i80+M47U5m7ZjFLT/fos6tIlLtKYyIlCObzcY1za5hQu8JRPpHsv7Qeob8MoTk7GSrSxMRsYzCiIgFWtdqzYcXfUikfyQbkzZy6y+3kpSdZHVZIiKWUBgRsUiTiCZ83OtjagbUZNPhTdz6860cyjpkdVkiIuVOo2lELLYtZRu3/nwrB7IOUCeoDqfXOJ0wvzBC3aGE+YXRO643dYPrWl2miEiJaWivSCWyI3UHt/x8C/sz9x9zW4RfBF/2/ZLooGgLKhMROXkKIyKVTEpOCosSFpGSk2JuuSnM3z2f7anbaVu7LR/2+hCX3WV1mSIixaYwIlIF7EzdybU/Xkt6XjqDWw5mZPuRVpckIlJsmvRMpAqoH1qfp895GoCP133MnJ1zLK5IRKT0KYyIVHAXNriQG1rcAMBjCx9jT/oeiysSESldTqsLEJH/NrLdSNYcWMOag2u4b859XNb4Mmw2G3abHTt2WtVqxek1Tre6TBGRk6I+IyKVxN70vVz949UnnD7+ssaXcV+7+6gZULOcKxMROT51YBWpgtYfWs/kDZPJ8eZgYOAzfGTkZbAoYREAwa5ghrUZxoDmA3Da1fApItZSGBGpRtYcWMPzS55n3aF1ADQJb8LL3V6mUVgjiysTkeqs2oURn89Hbm5uOVcmVZnL5cLhcFhdRrF5fV6mbp7KayteIzknmZigGCZdMonagbWtLk1EqqlqFUZyc3PZtm0bPp/PguqkKgsPDyc6OhqbzWZ1KcWWlJ3EoJmD2J66nWYRzZjQewLB7mCryxKRaqjahBHDMNi5cyd5eXnUqVMHu12jleXUGYZBZmYm+/fvJzw8nJiYGKtLKpHdabu5YcYNHMo+RKeYTrzd421cDs3eKiLlq7hhpNL3cPN4PGRmZlKnTh0CAwOtLkeqkICAAAD2799P7dq1K9Upm3oh9Xir51sM/mkwf+z9gycXP8mz5zxbqVp4RKT6qPTNCF6vFwC3221xJVIVFQTcvLw8iyspuZY1WvJy15dx2Bx8v+V7xiwdw6GsQ1aXJSJyjEofRgroG5+Uhcr+vjqv3nmM7jwagM83fk6Pr3pw16938dO2n8j2ZFtcnYiIqdKfphGRf9e/aX8CnAFMWj+JNQfXMH/PfObvmU+AM4AwvzDs2AtDV+3A2lwSdwkXx11MmF+YxZWLSHWhMFJJzJ07l+7du3P48GHCw8OtLkcqmYvjLubiuIvZlrKNH7f+yI9bfiQhI4EsT1aR/fak72Hl/pW8+OeLXFD/Ai5vfDld6nTBYa88/WVEpPJRGBGpRuLC4ri77d0MazOMrclbi8zk6jN8/HXwL6Ztnsamw5v4efvP/Lz9Z1rVbMXz5z5Pw7CGVpcvIlWUwohINWS32WkS0eSY69vUbsMNLW5gY9JGpm2exvdbvuevg39x9Q9XM7L9SAY0G1Dp+9GISMVTZTqwVjbdunXj7rvvZsSIEURERBAVFcUHH3xARkYGgwcPJiQkhCZNmjBz5szj3n/ChAmEh4czbdo0mjZtir+/P7169WLXrl3l/EqkqrHZbLSo0YJRHUcx9fKpdIzpSLY3m+eXPM8dv97Bvox9VpcoIlVMlQsjhmGQmeuxZCvp/HETJ06kZs2aLF26lLvvvps777yTq6++mi5durBixQouuugibrzxRjIzM497/8zMTJ577jk++eQTFi5cSHJyMgMGDCiNwygCQHRQNO9f+D6PdHgEP4cfixIW0f/7/sQnxVtdmohUIZV+Btbs7Gy2bdtGXFwc/v7+ZOZ6OH30z5bUuf7pXgS6i3fmq1u3bni9XubPnw+Y86WEhYXRv39/PvnkEwASExOJiYlh8eLFZGdnF+nAOmHCBAYPHswff/xBx44dAdi4cSMtWrRgyZIldOjQoWxeZDXzz/dXdbY1ZSuP/P4IG5I2EB0UzeRLJlMrsJbVZYlIBVbcGVirXMtIZXLmmWcW/u5wOKhRowatWrUqvC4qKgowZwA9HqfTydlnn114uXnz5oSHh7Nhw4Yyqliqs0Zhjfjgog9oGNqQxIxE7vntHs1VIiKlosp1YA1wOVj/dC/LnrskXK6ia4XYbLYi1xV0FNQCgFJRhPmF8VaPt7h+xvWsPbSWxxY8xktdX8Ju0/caETl5VS6M2Gy2Yp8qqew8Hg/Lli0rPCUTHx9PcnIyLVq0sLgyqcrqh9bnlW6vcNus2/hlxy80XNWQu9veDZh9tlJyUkjJTSHEHUKYO0xzlIjIf6oen9pVlMvl4u677+b111/H6XQyfPhwOnXqpP4iUubOjj6b0Z1GM3rRaN5f8z6rD6zmUNYhEtITyPQU7XAd6g4l3C+cM2qewbXNrqVt7bYaHiwiRZxU2+pbb71Fw4YN8ff3p2PHjixduvSE+06YMAGbzVZkq+4dAUtLYGAgDz/8MNdffz3nnHMOwcHBTJkyxeqypJq4oukV3HLGLQAs2buEzcmbC4NIoPPICtqpuansTNvJjG0zGPTTIK7+4Wq+3vQ1mXnHHyUmItVPiVtGpkyZwsiRI3n33Xfp2LEjr776Kr169SI+Pp7atWsf9z6hoaHExx8ZCqhvReb07v+0ffv2Y647erDT8QY+9e/fn/79+5dmaSLFdu9Z99IorBF5vjzqBNUhJjiGmKAY/J3+eHwe85RNTgoHsg4wc9tMpm+dTvzheJ5a/BTjlo/j+ubXc+PpN2odHJFqrsRhZNy4cQwdOpTBgwcD8O677zJ9+nQ++ugjHnnkkePex2azER0dfWqVikiFY7fZubzJ5ce9zWl3UiOgBjUCatAovBEdYzpyX7v7mLZ5Gl9s/ILd6bt5b817TNowqTCURPhHlPMrEJGKoESnaXJzc1m+fDk9e/Y88gB2Oz179mTx4sUnvF96ejoNGjQgNjaWyy+/nHXr1v3r8+Tk5JCamlpkE5HKL8wvjEEtBzG9/3TGdRtHs4hmZORl8MFfH9Drm168sPQFVu1fhc/QCDKR6qREYeTgwYN4vd7C+S8KREVFkZiYeNz7NGvWjI8++ojvvvuOSZMm4fP56NKlC7t37z7h84wZM4awsLDCLTY2tiRlVgs333wzycnJVpchclLsNjsXNriQL/t+yWvdX6NFZAuyPFlM2jCJG2feSM+vevLsH8+yZO8SvD6v1eWKSBkr0QysCQkJ1K1bl0WLFtG5c+fC6x966CHmzZvHkiVL/vMx8vLyaNGiBddddx3PPPPMcffJyckhJyen8HJqaiqxsbHFmoFVpDTp/VU+DMNgwZ4FTN82nXm75pGel154W52gOlx12lVc0fQKagbUtLBKESmp4s7AWqI+IzVr1sThcLBvX9GFsvbt21fsPiEul4u2bduyefPmE+7j5+eHn59fSUoTkUrMZrNxXr3zOK/eeeR6c/lj7x/8uuNXft35KwkZCby+8nXeXvU23et35/LGl9OqVisi/SOtLltESkmJwojb7aZdu3bMnj2bfv36AebsoLNnz2b48OHFegyv18tff/3FJZdcUuJiRaTqczvcnF/vfM6vdz6PdnyUX3b8wpfxX7L6wGpm7ZjFrB2zAIgJiqFljZa0rNmy8Geo+8TfvESk4irxaJqRI0cyaNAg2rdvT4cOHXj11VcLl70HuOmmm6hbty5jxowB4Omnn6ZTp040adKE5ORkXnrpJXbs2MGQIUNK95WISJXj7/TnssaXcVnjy4hPiufrTV/zx94/2J66nb0Ze9mbsZdfd/5auH/D0Ia0qtmKs6PP5rLGl2n2V5FKosRh5Nprr+XAgQOMHj2axMRE2rRpw08//VTYqXXnzp3Y7Uf6xR4+fJihQ4eSmJhIREQE7dq1Y9GiRZx++uml9ypEpMprFtmMxzo9BkBabhobkzay/tB61h5cy9qDa9mdvpvtqdvZnrqdH7b+wA9bf+CF817QysIilUCJOrBa5d86wKiDoZQlvb8qj8PZh1l7cC2rDqzi0/WfkuXJItI/krHnjaVznc7//QAiUuqK24FVS21WEnPnzsVms1k+nPfJJ5+kTZs2ltYgcjwR/hGcV+887m57N1MunULTiKYkZSdx+6zbeXPlm/86RDg+KZ57fruHF/98kVxvbjlWLSKgMCIiVVBcWByTL5nMVaddhYHBe2ve46ofruKbTd+Q7cku3C89N50Xlr7AtT9ey5xdc/h0/acM/WUoSdlJFlYvUv0ojAhgzq4rUpX4O/15ovMTjD1vLEGuIDYnb+bJxU9y4dcX8tqK15j691Qum3YZkzZMwmt4Oa/ueQS7glmxfwXXT7+evw//bfVLEKk2FEYs0q1bN+6++25GjBhBREQEUVFRfPDBB4Ujk0JCQmjSpAkzZ8487v0nTJhAeHg406ZNo2nTpvj7+9OrVy927dpVrOcvON0yfvz4Iv0hkpOTGTJkCLVq1SI0NJQLLriA1atX/+vrGDFiRJHr+vXrx80331ysOkTKWp9Gffj5yp+5v9391AmqQ3JOMuP/Gs/oRaM5kHWABqENeLfnu7zd820+u+QzYkNi2ZO+hxtm3MC8XfOsLl+kWijxaJoKzzDAqqXJXYFQghWJJ06cyEMPPcTSpUuZMmUKd955J1OnTuWKK67g0Ucf5ZVXXuHGG29k586dx71/ZmYmzz33HJ988glut5u77rqLAQMGsHDhwmI9/+bNm/nmm2/49ttvcTjMIZBXX301AQEBzJw5k7CwMN577z169OjBpk2biIzUJFNSOYX5hXHzGTdz4+k3MnfXXCZtmMTGpI3c3PJmBp8xGLfDDUCj8EZMvmQyI+eN5M/EPxn+23Cig6JpENKA+qH1aRDagFY1W9G2dlutPi5SiqpeGMnLhOfrWPPcjyaAO6jYu7du3Zr//e9/AIwaNYqxY8dSs2ZNhg4dCsDo0aN55513WLNmzXHvn5eXx5tvvknHjh0BM9y0aNGCpUuX0qFDh/98/tzcXD755BNq1TKHPi5YsIClS5eyf//+whlw/+///o9p06bx9ddfc9tttxX7tYlURA67gx4NetCjQQ8MwzhuoAj3D+e9C9/jhaUvMCV+CokZiSRmJLIk8chyF/WC63FZ48vo27gv9ULqledLEKmSql4YqUTOPPPMwt8dDgc1atSgVatWhdcVzN2yf//+4w6JcjqdnH322YWXmzdvTnh4OBs2bChWGGnQoEFhEAFYvXo16enp1KhRo8h+WVlZbNmypfgvTKQS+LeWDZfdxf86/Y/hbYazPXU7O1J3sCN1B9tTt7MoYRG703fz9uq3eXv127St3ZYgVxCZeZlkejLJyMsgKjCKu9vezVlRZ5XjKxKpvKpeGHEFmi0UVj13SXZ3uYpcttlsRa4r+GPp85XNcupBQUVbcdLT04mJiWHu3LnH7BseHn7cx7Db7fxzqpq8vLzSKlHEUuH+4bTxb0Ob2m0Kr8vyZDF752y+2/wdS/YuYeX+lcfcb1faLgb9NIiL4y5mZLuRRAcVb+0ukeqq6oURm61Ep0oqM4/Hw7JlywpbQeLj40lOTqZFixYn9XhnnXUWiYmJOJ1OGjZsWKz71KpVi7179xZe9nq9rF27lu7du59UDSIVXYAzgEsbXcqljS4lMSORhXsWYrfZCXIFEeQKwt/pz/St0/l609fM3DaTubvmcssZt3BaxGkcyj7EoaxDHMw6CEDzyOa0qtmKxuGNcdqr3p9jkeLSu78Sc7lc3H333bz++us4nU6GDx9Op06dinWK5nh69uxJ586d6devHy+++CKnnXYaCQkJTJ8+nSuuuIL27dsfc58LLriAkSNHMn36dBo3bsy4ceMsn5hNpLxEB0Vz5WlXHnN9u6h2XH3a1YxZOoaV+1fy1qq3/vVx/B3+NI9sTmxILDUCahDpH0mkfyS1AmvRNLwpNQNqqsOsVGkKI5VYYGAgDz/8MNdffz179uzhvPPO48MPPzzpx7PZbMyYMYPHHnuMwYMHc+DAAaKjozn//PML+6/80y233MLq1au56aabcDqd3HfffWoVEQFa1GjBxN4TmbFtBpPWTwKgZkDNwrDhMTysP7iedYfWkZ6XzqoDq1h1YNVxHyvCL4LTIk+jWUQzusd2p330sV8MRCozrU1TSU2YMIERI0aoFaKMVdf3l5Qfn+FjR+oO1h9az77MfSRlJZGUncSh7EMkpCewM20nPqNov7E+jfrwYPsHqRFQ4wSPKlIxFHdtGrWMiIhYyG6zExcWR1xY3HFvz/ZksyV5C/GH41m+bzk/bPmB6Vun8/vu3xnZbiT9m/bHbjPnr0zJSWF3+m5cdheNwhqpH4pUGnqnVlEtW7Zkx44dx73tvffeY+DAgeVckYicDH+nPy1rtqRlzZb0b9qf65pfx9OLn2ZD0gaeWvwUX2z8Apfdxc60naTmphbez8/hR7OIZrSo0YKWNVpybt1zqRVY61+eScQ6Ok1TRe3YseOEQ2yjoqIICQkp54oqJ72/pCLy+DxM3jCZN1e9SZYnq8htNQNqkuXJIiMvo8j1Nmy0rd2WixpeRI/6PagVUIsdqTtYd2gdaw+uZWvKVppGNOWiBhdxZq0zC1tbRE5FcU/TKIyI/Au9v6QiS8xIZP6e+UT6RVIvpB6xIbEEugLxGT52pu5k/aH1rD+0npX7V7LmYNGZnAOcAccEmQJRgVFc2OBCesf15syaZ2okj5w0hRGRUqD3l1QViRmJ/LrjV37Z8UvhRG0BzgCaRzanZY2WxIXFsXzfcubumkum58j6Xo3DGnPlaVfSt1Ffwv3DC6/PzMsk/nA8qTmpdIjpQIAzoJxfkVQGCiMipUDvL6mKDmQeIDU3lQahDY7p5JrjzWHhnoX8suMXftv5W2HridvupkeDHjhtTtYfWs+21G2Fo3xC3CFc1vgyrj7tahqHNz7h8+b58vht5298t/k7/Bx+3HLGLbSq1eqE+0vlpzAiUgr0/pLqLC03jRlbZ/D131+zMWnjMbfXCqiFw+4gMSOx8Lqzap9Ft9hu1A2uW7h5DA9fbfqKr+O/Zn/W/iKP0a1eN4a1HUbzyOZl/nqk/CmMiJQCvb9EwDAM1h9az0/bfyLQGcjpNU7n9BqnUyuwFj7Dx+KExXwZ/yXzds/Da3j/9bFq+NfgytOuZF/GPn7Y+kNh68oFsRfQIKwBPp8PHz4Mw8BrePH6vHgNLz7Dh91mp1fDXnSu07k8XraUAoURkVKg95dI8e3L2MePW39k0+FN7EnfQ0J6AgeyDgDQplYbrmt+HRc2uBCXw1wQdFvKNt5Z/Q4/bfsJg+J/FF3W+DIebP9gkT4sAF6fl83Jm6kTXIcQt0YMVgQKI1XM3Llz6d69O4cPHz7hCrrl4cknn2TatGmsWrWqVB6vYcOGjBgxghEjRpTK45W26vL+Eikr2Z5sMvIy/nW22L8P/82MbTPI8+Zht9mx2Ww4bA5sNhtOmxO7zY7D7mB32m6+/ftbDAwi/SN5pMMj9G7YmzUH1zBz20x+3v4zB7MOEuoOZVibYVzT7Jpj+sQkpCfw3ebvqBdSj0sbXXpKI4WSspPI9eZqVeZ/oRlYRUTEcv5Of/yd/x7km0Y05d6Ie4v1eFc0vYInFz3J5uTNPPT7Qzy/5HmSc5ILb3fanKTmpjJm6Ri+2vQVj3R4hA7RHVi+bzmTN05m9s7ZhaeGvtv8HU90foLY0NgSvSavz8sX8V/w2orXyPPl8WD7B7mu+XUaAn0KFEYEgNzcXNxut9VlWKI6v3aRyqZ1rdZ8eemXjP9rPO//9T7JOckEOAPoHtudi+MuplNMJ77b/B1vrHqDzcmbGfLLEOoE1SEhI6HwMc6qfRbrD61nSeIS+n/fn7va3MWNp99YrOnztyZvZfSi0aw+sLrwujFLx7D6wGqe6PwEga7AwusNw2DT4U2sO7SOtNw0UnNTSc9NJ8uTRdd6XenRoEfpHpxKTFPsWaRbt27cfffdjBgxgoiICKKiovjggw/IyMhg8ODBhISE0KRJE2bOnHnc+0+YMIHw8HCmTZtG06ZN8ff3p1evXuzatatYz//kk0/Spk0bxo8fX+QURHJyMkOGDKFWrVqEhoZywQUXsHr16hM+Trdu3Y45xdKvXz9uvvnmYtXxT+PGjaNVq1YEBQURGxvLXXfdRXp6OgAZGRmEhoby9ddfF7nPtGnTCAoKIi0tDYBdu3ZxzTXXEB4eTmRkJJdffjnbt28v3P/mm2+mX79+PPfcc9SpU4dmzZqdVK0iYg2Xw8Wdbe7kh34/8F7P95h7zVxeOP8FusV2w9/pz7XNr2X6FdO5rvl12G12EjIS8HP4cdVpVzH1sqlMvHgi317+LZ1iOpHtzWbc8nEM+HEAry5/lal/T2XFvhUczDpIZl4m+zL2sfnwZlbsW8E7q9/hqh+uYvWB1QS5gni80+M82P5BHDYHM7bNYOCMgWxP2c6+jH18tPYj+n/fn6t+uIonFj3B/y37P95f8z6TN05m6uapjJg7gscWPHbMTLnVVZVrGTEM44SzCpa1AGdAiZrpJk6cyEMPPcTSpUuZMmUKd955J1OnTuWKK67g0Ucf5ZVXXuHGG29k586dx71/ZmYmzz33HJ988glut5u77rqLAQMGsHDhwmI9/+bNm/nmm2/49ttvcTgcAFx99dUEBAQwc+ZMwsLCeO+99+jRowebNm0iMjKy2K/tZNntdl5//XXi4uLYunUrd911Fw899BBvv/02QUFBDBgwgI8//pirrrqq8D4Fl0NCQsjLy6NXr1507tyZ+fPn43Q6efbZZ+nduzdr1qwpbAGZPXs2oaGhzJo1q8xfk4iUjXoh9agXUu+4t4X5hfFox0e55rRrWH1gNT3q9yjS4TU2JJb3L3yf77Z8x0t/vkT84XjiD8cX63nPrXsuT3R+orCvSMuaLXlg3gNsTt7MVT9cRa43t7BDrsvuon1Ue2oE1CDYFUyIO4TU3FS+2vQV32/5npX7VzL2vLGcWevMEr32lJwUVuxbQe2g2rSs0bJE962IqlwYyfJk0XFyR0uee8n1S4o00f2X1q1b87///Q+AUaNGMXbsWGrWrMnQoUMBGD16NO+88w5r1qw57v3z8vJ488036djRfL0TJ06kRYsWLF26lA4dOvzn8+fm5vLJJ59Qq5a5eNaCBQtYunQp+/fvx8/PD4D/+7//Y9q0aXz99dfcdtttxX5tJ+voVpaGDRvy7LPPcscdd/D2228DMGTIELp06cLevXuJiYlh//79zJgxg19//RWAKVOm4PP5GD9+fGEw/PjjjwkPD2fu3LlcdNFFAAQFBTF+/HidnhGp4ppENKFJRJPj3maz2ejXpB/n1j2Xn7f/zPaU7exI3cHOtJ0kpCdgYOC0OQlxhxDiDiHCP4Jrm117TMfXdlHt+PLSL3lg3gOs2L8CME8F9W3cl4saXkSo+9iOmxfHXcyo+aPYlbaLm2bexNAzh9KrQS/iwuJw2B1F9jUMg32Z+1h3aB3LEpfxZ+KfbDq8CQMDu83Ow2c/zPUtri/Fo1b+qlwYqUzOPPNIEnY4HNSoUYNWrY7MRhgVFQXA/v37j9sL2el0cvbZZxdebt68OeHh4WzYsKFYYaRBgwaFQQRg9erVpKenU6NG0V7vWVlZbNmypfgv7BT8+uuvjBkzho0bN5KamorH4yE7O5vMzEwCAwPp0KEDLVu2ZOLEiTzyyCNMmjSJBg0acP755xe+hs2bNx+zEGB2dnaR19CqVSsFEREBzMUFB7YoupJ5jjcHr89b7BbvWoG1GN9rPAt2L6BpRNMTttgUaBfVjq8v+5pnFz/LzO0zeXf1u7y7+l0CnAG0iGzB6TVOJ8ebw+bkzWw+vJm0vLRjHiMqMIp9mfsYs3QM21K28XCHh4vV7+V4vD4vNpvNsgUSq1wYCXAGsOT6JZY9d0m4XK4il202W5HrCv4D+Hy+Uy/uOIKCgopcTk9PJyYmhrlz5x6z74mGE9vtdv45OvxEqwX/l+3bt3PppZdy55138txzzxEZGcmCBQu49dZbyc3NJTDQbHUaMmQIb731Fo888ggff/wxgwcPLjxW6enptGvXjs8+++yYxz86eP3ztYuIHM3P4QeO/97vaC67i+71uxd7/1B3KC+c/wJdY7vyZfyXbEjaQJYnixX7VxS2sBRw2pw0DGtIu6h2tI9ub5768a/Bx+s+5tXlr/JF/BfsTNvJ/3X9v+POsZKWm8b6Q+tZe3At8UnxJOUkkZqTSmpuKqk5qaTlpfF1369pFmlNH7oqF0ZsNluJTpVUZh6Ph2XLlhW2gsTHx5OcnEyLFi1O6vHOOussEhMTcTqdNGzYsFj3qVWrFnv37i287PV6Wbt2Ld27F/8/ZIHly5fj8/l4+eWXsdvNdP7ll18es98NN9zAQw89xOuvv8769esZNGhQkdcwZcoUateu/a9j2kVEKgKbzUafRn3o06gPXp+XHak7WHdoHesPrcff6U/T8KY0iWhCXGhc4WRxR7vljFtoENqAUfNHsShhEddPv552Ue3I9maT48kh25vN7rTdbE/d/p+1pOamlsErLJ4qF0aqE5fLxd13383rr7+O0+lk+PDhdOrUqVinaI6nZ8+edO7cmX79+vHiiy9y2mmnkZCQwPTp07niiito3779Mfe54IILGDlyJNOnT6dx48aMGzeO5OTkk3r+Jk2akJeXxxtvvEHfvn1ZuHAh77777jH7RURE0L9/fx588EEuuugi6tU70hw6cOBAXnrpJS6//HKefvpp6tWrx44dO/j222956KGHiuwrIlKROOwOGoU3olF4I/o27lvs+/Wo34OJvScy/LfhbE/dfsLgUTe4LqfXOJ2WNVoSFRRFqDvU3PxCCXOHEeYXVkqvpOQURiqxwMBAHn74Ya6//nr27NnDeeedx4cffnjSj2ez2ZgxYwaPPfYYgwcP5sCBA0RHR3P++ecX9l/5p1tuuYXVq1dz00034XQ6ue+++06qVQTMDr3jxo3jhRdeYNSoUZx//vmMGTOGm2666Zh9b731ViZPnswtt9xS5PrAwEB+//13Hn74Yfr3709aWhp169alR48eaikRkSqrRY0WTLl0Ct9t/g6Pz4O/0x8/hx/+Tn9q+NegZc2WRPqX/YjIk6Xp4CupCRMmMGLEiJNuhajsPv30U+677z4SEhLKtCNqdX1/iYiUBk0HL1VSZmYme/fuZezYsdx+++0aESMiUgVoBtYqqmXLlgQHBx93O95Ik7Iwf/78E9YQHBx8Uo/54osv0rx5c6Kjoxk1alQpVywiIlbQaZoqaseOHSccYhsVFXXMPBxlISsriz179pzw9iZNjj8RUUWi95eIyMnTaZpqrkGDBlaXQEBAQKUIHCIiYi2dphERERFLVZkwUgnONkklVFaz34qIyBGV/jSNy+XCZrNx4MABatWqVaJVc0VOxDAMcnNzOXDgAHa7XaN2RETKUKUPIw6Hg3r16rF79262b99udTlSxQQGBlK/fv3C6elFRKT0VfowAhAcHEzTpk1PeoE2keNxOBw4nU61tomIlLEqEUbA/OBwOEq4xKKIiIhYTm3PIiIiYimFEREREbGUwoiIiIhYqlL0GSmYQyQ1NdXiSkRERKS4Cj63/2susEoRRtLS0gCIjY21uBIREREpqbS0NMLCwk54e6VYKM/n85GQkEBISEipDrNMTU0lNjaWXbt2/esCPnLqdKzLj451+dLxLj861uWntI61YRikpaVRp06df52vqVK0jNjtdurVq1dmjx8aGqo3djnRsS4/OtblS8e7/OhYl5/SONb/1iJSQB1YRURExFIKIyIiImKpah1G/Pz8eOKJJ/Dz87O6lCpPx7r86FiXLx3v8qNjXX7K+1hXig6sIiIiUnVV65YRERERsZ7CiIiIiFhKYUREREQspTAiIiIilqrWYeStt96iYcOG+Pv707FjR5YuXWp1SZXemDFjOPvsswkJCaF27dr069eP+Pj4IvtkZ2czbNgwatSoQXBwMFdeeSX79u2zqOKqYezYsdhsNkaMGFF4nY5z6dqzZw833HADNWrUICAggFatWrFs2bLC2w3DYPTo0cTExBAQEEDPnj35+++/Lay4cvJ6vTz++OPExcUREBBA48aNeeaZZ4qsbaJjfXJ+//13+vbtS506dbDZbEybNq3I7cU5rklJSQwcOJDQ0FDCw8O59dZbSU9PP/XijGrqiy++MNxut/HRRx8Z69atM4YOHWqEh4cb+/bts7q0Sq1Xr17Gxx9/bKxdu9ZYtWqVcckllxj169c30tPTC/e54447jNjYWGP27NnGsmXLjE6dOhldunSxsOrKbenSpUbDhg2NM88807j33nsLr9dxLj1JSUlGgwYNjJtvvtlYsmSJsXXrVuPnn382Nm/eXLjP2LFjjbCwMGPatGnG6tWrjcsuu8yIi4szsrKyLKy88nnuueeMGjVqGD/++KOxbds246uvvjKCg4ON1157rXAfHeuTM2PGDOOxxx4zvv32WwMwpk6dWuT24hzX3r17G61btzb++OMPY/78+UaTJk2M66677pRrq7ZhpEOHDsawYcMKL3u9XqNOnTrGmDFjLKyq6tm/f78BGPPmzTMMwzCSk5MNl8tlfPXVV4X7bNiwwQCMxYsXW1VmpZWWlmY0bdrUmDVrltG1a9fCMKLjXLoefvhh49xzzz3h7T6fz4iOjjZeeumlwuuSk5MNPz8/4/PPPy+PEquMPn36GLfcckuR6/r3728MHDjQMAwd69LyzzBSnOO6fv16AzD+/PPPwn1mzpxp2Gw2Y8+ePadUT7U8TZObm8vy5cvp2bNn4XV2u52ePXuyePFiCyurelJSUgCIjIwEYPny5eTl5RU59s2bN6d+/fo69idh2LBh9OnTp8jxBB3n0vb999/Tvn17rr76amrXrk3btm354IMPCm/ftm0biYmJRY53WFgYHTt21PEuoS5dujB79mw2bdoEwOrVq1mwYAEXX3wxoGNdVopzXBcvXkx4eDjt27cv3Kdnz57Y7XaWLFlySs9fKRbKK20HDx7E6/USFRVV5PqoqCg2btxoUVVVj8/nY8SIEZxzzjmcccYZACQmJuJ2uwkPDy+yb1RUFImJiRZUWXl98cUXrFixgj///POY23ScS9fWrVt55513GDlyJI8++ih//vkn99xzD263m0GDBhUe0+P9TdHxLplHHnmE1NRUmjdvjsPhwOv18txzzzFw4EAAHesyUpzjmpiYSO3atYvc7nQ6iYyMPOVjXy3DiJSPYcOGsXbtWhYsWGB1KVXOrl27uPfee5k1axb+/v5Wl1Pl+Xw+2rdvz/PPPw9A27ZtWbt2Le+++y6DBg2yuLqq5csvv+Szzz5j8uTJtGzZklWrVjFixAjq1KmjY12FVcvTNDVr1sThcBwzsmDfvn1ER0dbVFXVMnz4cH788UfmzJlDvXr1Cq+Pjo4mNzeX5OTkIvvr2JfM8uXL2b9/P2eddRZOpxOn08m8efN4/fXXcTqdREVF6TiXopiYGE4//fQi17Vo0YKdO3cCFB5T/U05dQ8++CCPPPIIAwYMoFWrVtx4443cd999jBkzBtCxLivFOa7R0dHs37+/yO0ej4ekpKRTPvbVMoy43W7atWvH7NmzC6/z+XzMnj2bzp07W1hZ5WcYBsOHD2fq1Kn89ttvxMXFFbm9Xbt2uFyuIsc+Pj6enTt36tiXQI8ePfjrr79YtWpV4da+fXsGDhxY+LuOc+k555xzjhmivmnTJho0aABAXFwc0dHRRY53amoqS5Ys0fEuoczMTOz2oh9NDocDn88H6FiXleIc186dO5OcnMzy5csL9/ntt9/w+Xx07Njx1Ao4pe6vldgXX3xh+Pn5GRMmTDDWr19v3HbbbUZ4eLiRmJhodWmV2p133mmEhYUZc+fONfbu3Vu4ZWZmFu5zxx13GPXr1zd+++03Y9myZUbnzp2Nzp07W1h11XD0aBrD0HEuTUuXLjWcTqfx3HPPGX///bfx2WefGYGBgcakSZMK9xk7dqwRHh5ufPfdd8aaNWuMyy+/XMNNT8KgQYOMunXrFg7t/fbbb42aNWsaDz30UOE+OtYnJy0tzVi5cqWxcuVKAzDGjRtnrFy50tixY4dhGMU7rr179zbatm1rLFmyxFiwYIHRtGlTDe09VW+88YZRv359w+12Gx06dDD++OMPq0uq9IDjbh9//HHhPllZWcZdd91lREREGIGBgcYVV1xh7N2717qiq4h/hhEd59L1ww8/GGeccYbh5+dnNG/e3Hj//feL3O7z+YzHH3/ciIqKMvz8/IwePXoY8fHxFlVbeaWmphr33nuvUb9+fcPf399o1KiR8dhjjxk5OTmF++hYn5w5c+Yc9+/zoEGDDMMo3nE9dOiQcd111xnBwcFGaGioMXjwYCMtLe2Ua7MZxlHT2omIiIiUs2rZZ0REREQqDoURERERsZTCiIiIiFhKYUREREQspTAiIiIillIYEREREUspjIiIiIilFEZERETEUgojIiIiYimFEREREbGUwoiIiIhYSmFERERELPX/BTv2uK2E+6QAAAAASUVORK5CYII="
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "execution_count": 17
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T08:54:30.655253Z",
     "start_time": "2025-08-16T08:53:15.606494Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 模型过拟合\n",
    "model = nn.Sequential(\n",
    "    nn.Linear(784, 30, bias=False), nn.LayerNorm(30), nn.ReLU(),\n",
    "    nn.Linear( 30, 20, bias=False), nn.LayerNorm(20), nn.ReLU(),\n",
    "    nn.Linear( 20, 10)\n",
    ")\n",
    "\n",
    "loss['mlp_relu_layer_overfitting'] = train_model(model, optim.Adam(model.parameters(), lr=0.01), epochs=20)"
   ],
   "id": "a9ce60897007cec0",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 0 train 0.200 val 0.222 test 0.224\n",
      "epoch 1 train 0.125 val 0.158 test 0.158\n",
      "epoch 2 train 0.101 val 0.139 test 0.133\n",
      "epoch 3 train 0.108 val 0.140 test 0.139\n",
      "epoch 4 train 0.069 val 0.130 test 0.112\n",
      "epoch 5 train 0.072 val 0.127 test 0.126\n",
      "epoch 6 train 0.059 val 0.127 test 0.123\n",
      "epoch 7 train 0.066 val 0.118 test 0.118\n",
      "epoch 8 train 0.058 val 0.119 test 0.120\n",
      "epoch 9 train 0.056 val 0.129 test 0.116\n",
      "epoch 10 train 0.036 val 0.122 test 0.115\n",
      "epoch 11 train 0.064 val 0.150 test 0.130\n",
      "epoch 12 train 0.043 val 0.127 test 0.131\n",
      "epoch 13 train 0.034 val 0.146 test 0.116\n",
      "epoch 14 train 0.031 val 0.123 test 0.115\n",
      "epoch 15 train 0.037 val 0.158 test 0.125\n",
      "epoch 16 train 0.029 val 0.155 test 0.114\n",
      "epoch 17 train 0.039 val 0.156 test 0.145\n",
      "epoch 18 train 0.028 val 0.154 test 0.128\n",
      "epoch 19 train 0.022 val 0.151 test 0.127\n"
     ]
    }
   ],
   "execution_count": 18
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T08:55:50.898268Z",
     "start_time": "2025-08-16T08:55:50.893987Z"
    }
   },
   "cell_type": "code",
   "source": [
    "m = nn.Dropout(0.5)\n",
    "x = torch.randn(5, requires_grad=True)"
   ],
   "id": "6a824dfcbd49a252",
   "outputs": [],
   "execution_count": 19
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T08:55:54.924866Z",
     "start_time": "2025-08-16T08:55:54.919939Z"
    }
   },
   "cell_type": "code",
   "source": "x",
   "id": "a199a9574bf1410c",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-0.6472,  0.8302,  1.1706, -0.1630,  0.6412], requires_grad=True)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 20
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T08:55:58.756010Z",
     "start_time": "2025-08-16T08:55:58.748786Z"
    }
   },
   "cell_type": "code",
   "source": [
    "m.train()  # 最好指明\n",
    "l = m(x)\n",
    "l.sum().backward()\n",
    "l, x.grad"
   ],
   "id": "d7a5d7214b39dff2",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([-1.2943,  0.0000,  0.0000, -0.3261,  1.2824], grad_fn=<MulBackward0>),\n",
       " tensor([2., 0., 0., 2., 2.]))"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 21
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T08:56:04.806183Z",
     "start_time": "2025-08-16T08:56:04.801181Z"
    }
   },
   "cell_type": "code",
   "source": [
    "m.eval()  # 在评估模式下 dropout 不应该起作用，所以需要这个进行切换\n",
    "m(x)"
   ],
   "id": "a0fa1fb0842b13a1",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-0.6472,  0.8302,  1.1706, -0.1630,  0.6412], requires_grad=True)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 22
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T09:00:23.830453Z",
     "start_time": "2025-08-16T08:58:37.613746Z"
    }
   },
   "cell_type": "code",
   "source": [
    "model = nn.Sequential(\n",
    "    nn.Linear(784, 30, bias=False), nn.LayerNorm(30), nn.ReLU(), nn.Dropout(0.2),\n",
    "    nn.Linear( 30, 20, bias=False), nn.LayerNorm(20), nn.ReLU(), nn.Dropout(0.2),\n",
    "    nn.Linear( 20, 10)\n",
    ")\n",
    "\n",
    "loss['mlp_relu_layer_dropout'] = train_model(model, optim.Adam(model.parameters(), lr=0.01), epochs=20)"
   ],
   "id": "dd05187016562c2d",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 0 train 0.280 val 0.269 test 0.286\n",
      "epoch 1 train 0.190 val 0.208 test 0.221\n",
      "epoch 2 train 0.183 val 0.203 test 0.199\n",
      "epoch 3 train 0.154 val 0.193 test 0.169\n",
      "epoch 4 train 0.154 val 0.188 test 0.177\n",
      "epoch 5 train 0.130 val 0.183 test 0.162\n",
      "epoch 6 train 0.124 val 0.184 test 0.171\n",
      "epoch 7 train 0.119 val 0.163 test 0.163\n",
      "epoch 8 train 0.111 val 0.172 test 0.153\n",
      "epoch 9 train 0.126 val 0.173 test 0.184\n",
      "epoch 10 train 0.111 val 0.161 test 0.159\n",
      "epoch 11 train 0.108 val 0.161 test 0.149\n",
      "epoch 12 train 0.118 val 0.171 test 0.155\n",
      "epoch 13 train 0.101 val 0.151 test 0.139\n",
      "epoch 14 train 0.106 val 0.152 test 0.143\n",
      "epoch 15 train 0.109 val 0.164 test 0.146\n",
      "epoch 16 train 0.097 val 0.150 test 0.150\n",
      "epoch 17 train 0.103 val 0.160 test 0.162\n",
      "epoch 18 train 0.107 val 0.151 test 0.159\n",
      "epoch 19 train 0.091 val 0.146 test 0.160\n"
     ]
    }
   ],
   "execution_count": 24
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T09:02:23.200845Z",
     "start_time": "2025-08-16T09:00:43.053763Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 惩罚项\n",
    "model = nn.Sequential(\n",
    "    nn.Linear(784, 30, bias=False), nn.LayerNorm(30), nn.ReLU(),\n",
    "    nn.Linear( 30, 20, bias=False), nn.LayerNorm(20), nn.ReLU(),\n",
    "    nn.Linear( 20, 10)\n",
    ")\n",
    "\n",
    "_ = train_model(model, optim.Adam(model.parameters(), lr=0.01), epochs=20, penalty=True)"
   ],
   "id": "7ce5cf8751c47c49",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 0 train 0.252 val 0.274 test 0.260\n",
      "epoch 1 train 0.236 val 0.247 test 0.250\n",
      "epoch 2 train 0.234 val 0.239 test 0.229\n",
      "epoch 3 train 0.227 val 0.230 test 0.217\n",
      "epoch 4 train 0.222 val 0.235 test 0.241\n",
      "epoch 5 train 0.203 val 0.240 test 0.213\n",
      "epoch 6 train 0.201 val 0.210 test 0.196\n",
      "epoch 7 train 0.206 val 0.201 test 0.200\n",
      "epoch 8 train 0.212 val 0.237 test 0.216\n",
      "epoch 9 train 0.250 val 0.243 test 0.251\n",
      "epoch 10 train 0.182 val 0.225 test 0.204\n",
      "epoch 11 train 0.212 val 0.234 test 0.216\n",
      "epoch 12 train 0.193 val 0.204 test 0.214\n",
      "epoch 13 train 0.216 val 0.206 test 0.205\n",
      "epoch 14 train 0.231 val 0.213 test 0.207\n",
      "epoch 15 train 0.253 val 0.252 test 0.237\n",
      "epoch 16 train 0.200 val 0.218 test 0.207\n",
      "epoch 17 train 0.217 val 0.236 test 0.217\n",
      "epoch 18 train 0.200 val 0.215 test 0.221\n",
      "epoch 19 train 0.221 val 0.229 test 0.222\n"
     ]
    }
   ],
   "execution_count": 25
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "# 理解卷积\n",
    "\n",
    "来自唐一旦老师的视频 [徒手实现卷积神经网络--结构详解与代码实现](https://www.bilibili.com/video/BV1yx4y1b7c9) 。\n",
    "\n",
    "同时推荐观看飞天侠客的视频 [神经网络中永远也搞不明白的矩阵和CNN【白话DeepSeek04】](https://www.bilibili.com/video/BV1wXQhY8EJ6) 。\n",
    "\n",
    "视频中给出的工具：\n",
    "\n",
    "+ 卷积操作图示：https://setosa.io/ev/image-kernels/\n",
    "+ GPU 对比页面用 AI 工具生成的代码：https://www.blackbox.ai/\n",
    "+ 手写数字识别过程演示：https://felixzhang00.github.io/DL/mnist/\n",
    "+ 3D CNN 网络可视化：https://adamharley.com/nn_vis/cnn/3d.html"
   ],
   "id": "f0cfafcf1f3db4ff"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T09:19:10.840381Z",
     "start_time": "2025-08-16T09:19:10.832358Z"
    }
   },
   "cell_type": "code",
   "source": [
    "class CNN(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.conv1 = nn.Conv2d(1, 20, (5, 5))\n",
    "        self.pool1 = nn.MaxPool2d(2, 2)\n",
    "        self.conv2 = nn.Conv2d(20, 40, (5, 5))\n",
    "        self.pool2 = nn.MaxPool2d(2, 2)\n",
    "        self.fc1 = nn.Linear(40 * 4 * 4, 120)\n",
    "        self.fc2 = nn.Linear(120, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        # x : (B, 1, 28, 28)\n",
    "        B = x.shape[0]                       # (B,  1, 28, 28)\n",
    "        x = F.relu(self.conv1(x))            # (B, 20, 24, 24)\n",
    "        x = self.pool1(x)                    # (B, 20, 12, 12)\n",
    "        x = F.relu(self.conv2(x))            # (B, 40,  8,  8)\n",
    "        x = self.pool2(x)                    # (B, 40,  4,  4)\n",
    "        x = F.relu(self.fc1(x.view(B, -1)))  # (B, 120)\n",
    "        x = self.fc2(x)                      # (B, 10)\n",
    "        return x\n",
    "\n",
    "model = CNN()"
   ],
   "id": "9889abda7271ca73",
   "outputs": [],
   "execution_count": 31
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T09:25:12.973248Z",
     "start_time": "2025-08-16T09:25:12.968673Z"
    }
   },
   "cell_type": "code",
   "source": [
    "eval_iters = 10\n",
    "\n",
    "\n",
    "def estimate_loss(model):\n",
    "    re = {}\n",
    "    # 将模型切换为评估模式\n",
    "    model.eval()\n",
    "    re['train'] = _loss(model, train_loader)\n",
    "    re['val'] = _loss(model, val_loader)\n",
    "    re['test'] = _loss(model, test_loader)\n",
    "    # 将模型切换为训练模式\n",
    "    model.train()\n",
    "    return re\n",
    "\n",
    "\n",
    "@torch.no_grad()\n",
    "def _loss(model, dataloader):\n",
    "    # 估算模型效果\n",
    "    loss = []\n",
    "    acc = []\n",
    "    data_iter = iter(dataloader)\n",
    "    for t in range(eval_iters):\n",
    "        inputs, labels = next(data_iter)\n",
    "        # inputs: (500, 1, 28, 28)\n",
    "        # labels: (500)\n",
    "        B, C, H, W = inputs.shape\n",
    "        #logits = model(inputs.view(B, -1))\n",
    "        logits = model(inputs)\n",
    "        loss.append(F.cross_entropy(logits, labels))\n",
    "        # preds = torch.argmax(F.softmax(logits, dim=-1), dim=-1)\n",
    "        preds = torch.argmax(logits, dim=-1)\n",
    "        acc.append((preds == labels).sum() / B)\n",
    "    re = {\n",
    "        'loss': torch.tensor(loss).mean().item(),\n",
    "        'acc': torch.tensor(acc).mean().item()\n",
    "    }\n",
    "    return re"
   ],
   "id": "8a30c4880e8254dd",
   "outputs": [],
   "execution_count": 35
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T09:25:16.375059Z",
     "start_time": "2025-08-16T09:25:16.371054Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def train_model(model, optimizer, epochs=10, penalty=False):\n",
    "    lossi = []\n",
    "    for e in range(epochs):\n",
    "        for data in train_loader:\n",
    "            inputs, labels = data\n",
    "            #B, C, H, W = inputs.shape\n",
    "            #logits = model(inputs.view(B, -1))\n",
    "            logits = model(inputs)\n",
    "            loss = F.cross_entropy(logits, labels)\n",
    "            lossi.append(loss.item())\n",
    "            if penalty:\n",
    "                w = torch.cat([p.view(-1) for p in model.parameters()])\n",
    "                loss += 0.001 * w.abs().sum() + 0.002 * w.square().sum()\n",
    "            optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "        stats = estimate_loss(model)\n",
    "        train_loss = f'{stats[\"train\"][\"loss\"]:.3f}'\n",
    "        val_loss = f'{stats[\"val\"][\"loss\"]:.3f}'\n",
    "        test_loss = f'{stats[\"test\"][\"loss\"]:.3f}'\n",
    "        print(f'epoch {e} train {train_loss} val {val_loss} test {test_loss}')\n",
    "    return lossi"
   ],
   "id": "1764126f85a4368b",
   "outputs": [],
   "execution_count": 36
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T09:26:25.308249Z",
     "start_time": "2025-08-16T09:25:16.874778Z"
    }
   },
   "cell_type": "code",
   "source": "_ = train_model(model, optim.Adam(model.parameters(), lr=0.01))",
   "id": "3a5cc1c533cfc70f",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 0 train 0.040 val 0.062 test 0.042\n",
      "epoch 1 train 0.026 val 0.047 test 0.038\n",
      "epoch 2 train 0.028 val 0.048 test 0.035\n",
      "epoch 3 train 0.015 val 0.041 test 0.037\n",
      "epoch 4 train 0.016 val 0.047 test 0.039\n",
      "epoch 5 train 0.013 val 0.049 test 0.034\n",
      "epoch 6 train 0.037 val 0.075 test 0.050\n",
      "epoch 7 train 0.015 val 0.051 test 0.046\n",
      "epoch 8 train 0.013 val 0.068 test 0.055\n",
      "epoch 9 train 0.012 val 0.050 test 0.039\n"
     ]
    }
   ],
   "execution_count": 37
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T09:27:05.410738Z",
     "start_time": "2025-08-16T09:27:03.675745Z"
    }
   },
   "cell_type": "code",
   "source": "estimate_loss(model)",
   "id": "d6ccbd9c4066f059",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'train': {'loss': 0.010436898097395897, 'acc': 0.9973999857902527},\n",
       " 'val': {'loss': 0.042675744742155075, 'acc': 0.9891999959945679},\n",
       " 'test': {'loss': 0.04551061987876892, 'acc': 0.9873999357223511}}"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 38
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T09:30:20.301263Z",
     "start_time": "2025-08-16T09:30:20.295239Z"
    }
   },
   "cell_type": "code",
   "source": [
    "class CNN2(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.conv1 = nn.Conv2d(1, 20, (5, 5))\n",
    "        self.ln1 = nn.LayerNorm([20, 24, 24])   # 层级归一化，与 AlexNet 中的做法类似\n",
    "        self.pool1 = nn.MaxPool2d(2, 2)\n",
    "        self.conv2 = nn.Conv2d(20, 40, (5, 5))\n",
    "        self.ln2 = nn.LayerNorm([40, 8, 8])\n",
    "        self.pool2 = nn.MaxPool2d(2, 2)\n",
    "        self.fc1 = nn.Linear(40 * 4 * 4, 120)\n",
    "        self.dp = nn.Dropout(0.2)\n",
    "        self.fc2 = nn.Linear(120, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        # x : (B, 1, 28, 28)\n",
    "        B = x.shape[0]                       # (B,  1, 28, 28)\n",
    "        x = F.relu(self.ln1(self.conv1(x)))  # (B, 20, 24, 24)\n",
    "        x = self.pool1(x)                    # (B, 20, 12, 12)\n",
    "        x = F.relu(self.ln2(self.conv2(x)))  # (B, 40,  8,  8)\n",
    "        x = self.pool2(x)                    # (B, 40,  4,  4)\n",
    "        x = F.relu(self.fc1(x.view(B, -1)))  # (B, 120)\n",
    "        x = self.dp(x)\n",
    "        x = self.fc2(x)                      # (B, 10)\n",
    "        return x\n",
    "\n",
    "model2 = CNN2()"
   ],
   "id": "68f71de545000ef7",
   "outputs": [],
   "execution_count": 40
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T09:32:04.514629Z",
     "start_time": "2025-08-16T09:30:22.131375Z"
    }
   },
   "cell_type": "code",
   "source": "_ = train_model(model2, optim.Adam(model2.parameters(), lr=0.01))",
   "id": "51f88d5b595b4a9e",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 0 train 0.082 val 0.078 test 0.073\n",
      "epoch 1 train 0.032 val 0.045 test 0.046\n",
      "epoch 2 train 0.036 val 0.050 test 0.041\n",
      "epoch 3 train 0.025 val 0.046 test 0.037\n",
      "epoch 4 train 0.039 val 0.063 test 0.052\n",
      "epoch 5 train 0.018 val 0.044 test 0.039\n",
      "epoch 6 train 0.017 val 0.052 test 0.030\n",
      "epoch 7 train 0.014 val 0.045 test 0.031\n",
      "epoch 8 train 0.008 val 0.043 test 0.035\n",
      "epoch 9 train 0.016 val 0.059 test 0.042\n"
     ]
    }
   ],
   "execution_count": 41
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-16T09:32:25.678743Z",
     "start_time": "2025-08-16T09:32:24.477345Z"
    }
   },
   "cell_type": "code",
   "source": [
    "model2.eval()\n",
    "estimate_loss(model2)"
   ],
   "id": "8826cd0dfdb84773",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'train': {'loss': 0.011416323482990265, 'acc': 0.9964000582695007},\n",
       " 'val': {'loss': 0.04831470549106598, 'acc': 0.9893999099731445},\n",
       " 'test': {'loss': 0.03911430388689041, 'acc': 0.9908000230789185}}"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 42
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "# 残差链接\n",
    "\n",
    "来自唐一旦老师视频 [徒手实现卷积神经网络--残差网络（ResNet）](https://www.bilibili.com/video/BV1D1421r7sj)，同时你可以看笔记 7 ，这里代码的写法不一样。"
   ],
   "id": "d75bd1159d78df05"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-18T08:01:55.112813Z",
     "start_time": "2025-08-18T08:01:55.107802Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 卷积层的几个小技巧\n",
    "channels = torch.randint(1, 10, (1, ))  # low, high, size\n",
    "conv1 = nn.Conv2d(channels, channels, (3, 3), stride=1, padding=1)\n",
    "x = torch.randn(1, channels, 28, 28)\n",
    "print(x.shape, conv1(x).shape)"
   ],
   "id": "aaf83ecf4ba96787",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 2, 28, 28]) torch.Size([1, 2, 28, 28])\n"
     ]
    }
   ],
   "execution_count": 67
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-18T08:06:36.824793Z",
     "start_time": "2025-08-18T08:06:36.820480Z"
    }
   },
   "cell_type": "code",
   "source": [
    "class ResidualBlockSimplified(nn.Module):\n",
    "\n",
    "    def __init__(self, channels):\n",
    "        super().__init__()\n",
    "        self.covn1 = nn.Conv2d(channels, channels, (3, 3), stride=1, padding=1)\n",
    "        self.covn2 = nn.Conv2d(channels, channels, (3, 3), stride=1, padding=1)\n",
    "\n",
    "    def forward(self, x):\n",
    "        inputs = x\n",
    "        x = F.relu(self.covn1(x))\n",
    "        x = self.covn2(x)\n",
    "        # 残差连接\n",
    "        out = x + inputs\n",
    "        return F.relu(out)"
   ],
   "id": "d8ef43160a7e2b4e",
   "outputs": [],
   "execution_count": 68
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-18T08:06:38.065870Z",
     "start_time": "2025-08-18T08:06:38.055356Z"
    }
   },
   "cell_type": "code",
   "source": [
    "model = ResidualBlockSimplified(3)\n",
    "x = torch.randn(1, 3, 28, 28)\n",
    "model(x).shape"
   ],
   "id": "2e8ef4f93789a58d",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 3, 28, 28])"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 69
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-18T08:07:33.857559Z",
     "start_time": "2025-08-18T08:07:33.851635Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 这两个卷积操作的输出是一样形状的\n",
    "stride = torch.randint(1, 10, (1,))\n",
    "in_channels = torch.randint(1, 10, (1,))\n",
    "out_channels = torch.randint(1, 10, (1,))\n",
    "conv1 = nn.Conv2d(in_channels, out_channels, (3, 3), stride=stride, padding=1)\n",
    "conv2 = nn.Conv2d(in_channels, out_channels, (1, 1), stride=stride, padding=0)\n",
    "x = torch.randn(1, in_channels, 28, 28)\n",
    "conv1(x).shape, conv2(x).shape"
   ],
   "id": "a3a16e553db290c2",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([1, 1, 4, 4]), torch.Size([1, 1, 4, 4]))"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 70
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-18T08:19:21.960320Z",
     "start_time": "2025-08-18T08:19:21.953191Z"
    }
   },
   "cell_type": "code",
   "source": [
    "class ResidualBlock(nn.Module):\n",
    "\n",
    "    def __init__(self, in_channels, out_channels, stride=1):\n",
    "        super().__init__()\n",
    "        self.conv1 = nn.Conv2d(in_channels, out_channels, (3, 3), stride=stride, padding=1)\n",
    "        self.bn1 = nn.BatchNorm2d(out_channels)\n",
    "        self.conv2 = nn.Conv2d(out_channels, out_channels, (3, 3), stride=1, padding=1)\n",
    "        self.bn2 = nn.BatchNorm2d(out_channels)\n",
    "        self.downsample = None\n",
    "        if stride != 1 or in_channels != out_channels:\n",
    "            self.downsample = nn.Conv2d(in_channels, out_channels, (1, 1), stride=stride, padding=0)\n",
    "            self.bn3 = nn.BatchNorm2d(out_channels)\n",
    "\n",
    "    def forward(self, x):\n",
    "        inputs = x\n",
    "        x = F.relu(self.bn1(self.conv1(x)))\n",
    "        x = self.bn2(self.conv2(x))\n",
    "        if self.downsample is not None:\n",
    "            inputs = self.bn3(self.downsample(inputs))\n",
    "        out = x + inputs\n",
    "        out = F.relu(out)\n",
    "        return out"
   ],
   "id": "a3e5f070e8f8551e",
   "outputs": [],
   "execution_count": 71
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-08-18T08:19:23.791640Z",
     "start_time": "2025-08-18T08:19:23.786438Z"
    }
   },
   "cell_type": "code",
   "source": [
    "class ResNet(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.block1 = ResidualBlock(1, 20)\n",
    "        self.block2 = ResidualBlock(20, 40, stride=2)\n",
    "        self.block3 = ResidualBlock(40, 60, stride=2)\n",
    "        self.block4 = ResidualBlock(60, 80, stride=2)\n",
    "        self.block5 = ResidualBlock(80, 100, stride=2)\n",
    "        self.block6 = ResidualBlock(100, 120, stride=2)\n",
    "        self.fc = nn.Linear(120, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        # x : (B, 1, 28, 28)\n",
    "        B = x.shape[0]\n",
    "        x = self.block1(x)  # (B, 20, 28, 28)\n",
    "        x = self.block2(x)  # (B, 40, 14, 14)\n",
    "        x = self.block3(x)  # (B, 60,  7,  7)\n",
    "        x = self.block4(x)  # (B, 80,  4,  4)\n",
    "        x = self.block5(x)  # (B, 100, 2,  2)\n",
    "        x = self.block6(x)  # (B, 120, 1,  1)\n",
    "        x = self.fc(x.view(B, -1))\n",
    "        return x"
   ],
   "id": "147c1ccae12d0b25",
   "outputs": [],
   "execution_count": 72
  },
  {
   "metadata": {},
   "cell_type": "code",
   "outputs": [],
   "execution_count": null,
   "source": "",
   "id": "96bd25a37caded8e"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
